2025-04-02 03:57:59,886 [ 692919 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2025-04-02 03:57:59,887 [ 692919 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:97, check_args_and_update_paths) 2025-04-02 03:57:59,887 [ 692919 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:108, check_args_and_update_paths) 2025-04-02 03:57:59,887 [ 692919 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:110, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_yhqp7s --privileged --dns-search='.' --memory=30709035008 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=8b2301119731 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_postgresql_database_engine/test.py::test_datetime test_postgresql_database_engine/test.py::test_get_create_table_query_with_multidim_arrays test_postgresql_database_engine/test.py::test_inaccessible_postgresql_database_engine_filterable_on_system_tables test_postgresql_database_engine/test.py::test_postgres_database_engine_with_postgres_ddl test_postgresql_database_engine/test.py::test_postgres_database_old_syntax test_postgresql_database_engine/test.py::test_postgresql_database_engine_queries test_postgresql_database_engine/test.py::test_postgresql_database_engine_table_cache test_postgresql_database_engine/test.py::test_postgresql_database_engine_with_clickhouse_ddl test_postgresql_database_engine/test.py::test_postgresql_database_with_schema test_postgresql_database_engine/test.py::test_postgresql_fetch_tables test_postgresql_database_engine/test.py::test_postgresql_password_leak test_postgresql_database_engine/test.py::test_predefined_connection_configuration test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order test_prometheus_endpoint/test.py::test_prometheus_endpoint test_prometheus_protocols/test.py::test_64bit_id test_prometheus_protocols/test.py::test_create_as_table test_prometheus_protocols/test.py::test_custom_id_algorithm test_prometheus_protocols/test.py::test_default test_prometheus_protocols/test.py::test_external_tables test_prometheus_protocols/test.py::test_inner_engines test_prometheus_protocols/test.py::test_read_auth test_prometheus_protocols/test.py::test_remote_write_v1_status_code test_prometheus_protocols/test.py::test_tags_to_columns test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_read_only_table/test.py::test_restart_zookeeper test_recompression_ttl/test.py::test_recompression_multiple_ttls test_recompression_ttl/test.py::test_recompression_replicated test_recompression_ttl/test.py::test_recompression_simple test_recovery_time_metric/test.py::test_recovery_time_metric test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db test_refreshable_mv/test.py::test_refreshable_mv_in_system_db test_relative_filepath/test.py::test_filepath test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers test_reload_certificate/test.py::test_ECcert_reload test_reload_certificate/test.py::test_cert_with_pass_phrase test_reload_certificate/test.py::test_chain_reload test_reload_certificate/test.py::test_first_than_second_cert test_reload_clusters_config/test.py::test_add_cluster test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_simple_reload test_reload_clusters_config/test.py::test_update_one_cluster test_reloading_settings_from_users_xml/test.py::test_force_reload test_reloading_settings_from_users_xml/test.py::test_reload_on_timeout test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_enum test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_int test_reloading_settings_from_users_xml/test.py::test_unknown_setting_force_reload test_reloading_settings_from_users_xml/test.py::test_unknown_setting_reload_on_timeout 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain]' test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4]' test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format test_render_log_file_name_templates/test.py::test_check_file_names test_replica_can_become_leader/test.py::test_can_become_leader test_replica_is_active/test.py::test_replica_is_active test_replicated_zero_copy_projection_mutation/test.py::test_all_projection_files_are_dropped_when_part_is_dropped test_replicated_zero_copy_projection_mutation/test.py::test_hardlinks_preserved_when_projection_dropped test_replicating_constants/test.py::test_different_versions test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_restart_server/test.py::test_drop_memory_database test_restart_server/test.py::test_flushes_async_insert_queue test_restore_replica/test.py::test_restore_replica_alive_replicas test_restore_replica/test.py::test_restore_replica_invalid_tables test_restore_replica/test.py::test_restore_replica_parallel test_restore_replica/test.py::test_restore_replica_sequential test_rocksdb_read_only/test.py::test_dirctory_missing_after_stop test_rocksdb_read_only/test.py::test_read_only test_role/test.py::test_admin_option test_role/test.py::test_changing_default_roles_affects_new_sessions_only test_role/test.py::test_combine_privileges test_role/test.py::test_create_role test_role/test.py::test_function_current_roles test_role/test.py::test_grant_role_to_role test_role/test.py::test_introspection test_role/test.py::test_revoke_requires_admin_option 'test_role/test.py::test_role_expiration[False]' 'test_role/test.py::test_role_expiration[True]' test_role/test.py::test_roles_cache test_role/test.py::test_set_role test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable 'test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header]' test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated -vvv -ss" altinityinfra/integration-tests-runner:2165613c5fcd '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket= rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_postgresql_database_engine/test.py::test_datetime Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] Command:[docker ps | wc -l] test_reload_clusters_config/test.py::test_add_cluster test_s3_cluster/test.py::test_ambiguous_join test_restore_replica/test.py::test_restore_replica_alive_replicas test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] test_prometheus_protocols/test.py::test_64bit_id test_role/test.py::test_admin_option test_reloading_settings_from_users_xml/test.py::test_force_reload test_reload_certificate/test.py::test_ECcert_reload test_replication_credentials/test.py::test_credentials_and_no_credentials Stdout:1 No running containers Pruning Docker networks Stdout:1 Command:[docker network prune --force] No running containers Pruning Docker networks Command:[docker network prune --force] Stdout:1 Stdout:1 No running containers Pruning Docker networks No running containers Command:[docker network prune --force] Pruning Docker networks Command:[docker network prune --force] Stdout:1 No running containers Stdout:1 Pruning Docker networks No running containers Command:[docker network prune --force] Pruning Docker networks Command:[docker network prune --force] Stdout:1 No running containers Pruning Docker networks Command:[docker network prune --force] Stdout:1 No running containers Pruning Docker networks Command:[docker network prune --force] Stdout:1 Stdout:1 No running containers Pruning Docker networks Command:[docker network prune --force] No running containers Pruning Docker networks Command:[docker network prune --force] Stderr:Error response from daemon: a prune operation is already running Exitcode:1 Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Stderr:Error response from daemon: a prune operation is already running Exitcode:1 Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Stdout:net.ipv4.ip_local_port_range = 55000 65535 ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 Stdout:net.ipv4.ip_local_port_range = 55000 65535 ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 Running tests in /ClickHouse/tests/integration/test_replication_credentials/test.py ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true Cluster start called. is_up=False ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 ENV HOSTNAME e41643a81eb1 ENV SHLVL 0 ENV HOME /root ENV OLDPWD / ENV DOCKER_HELPER_TAG 5dc43a6382f0 ENV PYTHONUNBUFFERED 1 ENV DOCKER_PYTHON_BOTTLE_TAG caad4729259e ENV UBSAN_OPTIONS print_stacktrace=1 ENV PYTEST_ADDOPTS --dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_postgresql_database_engine/test.py::test_datetime test_postgresql_database_engine/test.py::test_get_create_table_query_with_multidim_arrays test_postgresql_database_engine/test.py::test_inaccessible_postgresql_database_engine_filterable_on_system_tables test_postgresql_database_engine/test.py::test_postgres_database_engine_with_postgres_ddl test_postgresql_database_engine/test.py::test_postgres_database_old_syntax test_postgresql_database_engine/test.py::test_postgresql_database_engine_queries test_postgresql_database_engine/test.py::test_postgresql_database_engine_table_cache test_postgresql_database_engine/test.py::test_postgresql_database_engine_with_clickhouse_ddl test_postgresql_database_engine/test.py::test_postgresql_database_with_schema test_postgresql_database_engine/test.py::test_postgresql_fetch_tables test_postgresql_database_engine/test.py::test_postgresql_password_leak test_postgresql_database_engine/test.py::test_predefined_connection_configuration test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order test_prometheus_endpoint/test.py::test_prometheus_endpoint test_prometheus_protocols/test.py::test_64bit_id test_prometheus_protocols/test.py::test_create_as_table test_prometheus_protocols/test.py::test_custom_id_algorithm test_prometheus_protocols/test.py::test_default test_prometheus_protocols/test.py::test_external_tables test_prometheus_protocols/test.py::test_inner_engines test_prometheus_protocols/test.py::test_read_auth test_prometheus_protocols/test.py::test_remote_write_v1_status_code test_prometheus_protocols/test.py::test_tags_to_columns test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_read_only_table/test.py::test_restart_zookeeper test_recompression_ttl/test.py::test_recompression_multiple_ttls test_recompression_ttl/test.py::test_recompression_replicated test_recompression_ttl/test.py::test_recompression_simple test_recovery_time_metric/test.py::test_recovery_time_metric test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db test_refreshable_mv/test.py::test_refreshable_mv_in_system_db test_relative_filepath/test.py::test_filepath test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers test_reload_certificate/test.py::test_ECcert_reload test_reload_certificate/test.py::test_cert_with_pass_phrase test_reload_certificate/test.py::test_chain_reload test_reload_certificate/test.py::test_first_than_second_cert test_reload_clusters_config/test.py::test_add_cluster test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_simple_reload test_reload_clusters_config/test.py::test_update_one_cluster test_reloading_settings_from_users_xml/test.py::test_force_reload test_reloading_settings_from_users_xml/test.py::test_reload_on_timeout test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_enum test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_int test_reloading_settings_from_users_xml/test.py::test_unknown_setting_force_reload test_reloading_settings_from_users_xml/test.py::test_unknown_setting_reload_on_timeout 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain]' test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4]' test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format test_render_log_file_name_templates/test.py::test_check_file_names test_replica_can_become_leader/test.py::test_can_become_leader test_replica_is_active/test.py::test_replica_is_active test_replicated_zero_copy_projection_mutation/test.py::test_all_projection_files_are_dropped_when_part_is_dropped test_replicated_zero_copy_projection_mutation/test.py::test_hardlinks_preserved_when_projection_dropped test_replicating_constants/test.py::test_different_versions test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_restart_server/test.py::test_drop_memory_database test_restart_server/test.py::test_flushes_async_insert_queue test_restore_replica/test.py::test_restore_replica_alive_replicas test_restore_replica/test.py::test_restore_replica_invalid_tables test_restore_replica/test.py::test_restore_replica_parallel test_restore_replica/test.py::test_restore_replica_sequential test_rocksdb_read_only/test.py::test_dirctory_missing_after_stop test_rocksdb_read_only/test.py::test_read_only test_role/test.py::test_admin_option test_role/test.py::test_changing_default_roles_affects_new_sessions_only test_role/test.py::test_combine_privileges test_role/test.py::test_create_role test_role/test.py::test_function_current_roles test_role/test.py::test_grant_role_to_role test_role/test.py::test_introspection test_role/test.py::test_revoke_requires_admin_option 'test_role/test.py::test_role_expiration[False]' 'test_role/test.py::test_role_expiration[True]' test_role/test.py::test_roles_cache test_role/test.py::test_set_role test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable 'test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header]' test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated -vvv -ss ENV CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge ENV COMPOSE_HTTP_TIMEOUT 600 ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ENV DOCKER_KERBERIZED_HADOOP_TAG latest ENV DOCKER_CHANNEL stable Stderr:Error response from daemon: a prune operation is already running ENV DOCKER_CLIENT_TIMEOUT 300 Exitcode:1 Stderr:Error response from daemon: a prune operation is already running ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Exitcode:1 ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 Stderr:Error response from daemon: a prune operation is already running Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e ENV PWD /ClickHouse/tests/integration Exitcode:1 ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 ENV CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config ENV TZ Etc/UTC ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] ENV DOCKER_BASE_TAG 8b2301119731 ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 ENV LC_CTYPE C.UTF-8 ENV INTEGRATION_TESTS_RUN_ID 0 ENV WORKER_FREE_PORTS 30200 30201 30202 30203 30204 30205 30206 30207 30208 30209 30210 30211 30212 30213 30214 30215 30216 30217 30218 30219 30220 30221 30222 30223 30224 30225 30226 30227 30228 30229 30230 30231 30232 30233 30234 30235 30236 30237 30238 30239 30240 30241 30242 30243 30244 30245 30246 30247 30248 30249 ENV PYTEST_XDIST_TESTRUNUID 63960e6506124119bf68bf57c6473af0 ENV PYTEST_XDIST_WORKER gw4 ENV PYTEST_XDIST_WORKER_COUNT 10 ENV PYTEST_CURRENT_TEST test_s3_cluster/test.py::test_ambiguous_join (setup) Stdout:net.ipv4.ip_local_port_range = 55000 65535 CLUSTER INIT base_config_dir:/clickhouse-config clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Setup Keeper Stdout:net.ipv4.ip_local_port_range = 55000 65535 Running tests in /ClickHouse/tests/integration/test_prometheus_protocols/test.py Stderr:Error response from daemon: a prune operation is already running Cluster name: project_name:roottests3cluster-gw4. Added instance name:s0_0_0 tag:8b2301119731 base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/.env', '--project-name', 'roottests3cluster-gw4', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Cluster start called. is_up=False Stderr:Error response from daemon: a prune operation is already running Exitcode:1 Cluster name: project_name:roottests3cluster-gw4. Added instance name:s0_0_1 tag:8b2301119731 base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/.env', '--project-name', 'roottests3cluster-gw4', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_1/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Running tests in /ClickHouse/tests/integration/test_postgresql_database_engine/test.py Cluster name: project_name:roottests3cluster-gw4. Added instance name:s0_1_0 tag:8b2301119731 base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/.env', '--project-name', 'roottests3cluster-gw4', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_1_0/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ Exitcode:1 Starting cluster... Cluster start called. is_up=False Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Stdout:net.ipv4.ip_local_port_range = 55000 65535 Running tests in /ClickHouse/tests/integration/test_s3_cluster/test.py Cluster start called. is_up=False Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Running tests in /ClickHouse/tests/integration/test_restore_replica/test.py Stderr:Error response from daemon: a prune operation is already running Cluster start called. is_up=False Exitcode:1 Stderr:Error response from daemon: a prune operation is already running Exitcode:1 Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Stdout:net.ipv4.ip_local_port_range = 55000 65535 Stdout:net.ipv4.ip_local_port_range = 55000 65535 Running tests in /ClickHouse/tests/integration/test_reload_clusters_config/test.py ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 Cluster start called. is_up=False ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 ENV HOSTNAME e41643a81eb1 Stdout:net.ipv4.ip_local_port_range = 55000 65535 ENV SHLVL 0 ENV HOME /root ENV OLDPWD / ENV DOCKER_HELPER_TAG 5dc43a6382f0 ENV PYTHONUNBUFFERED 1 ENV DOCKER_PYTHON_BOTTLE_TAG caad4729259e Running tests in /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/test.py ENV UBSAN_OPTIONS print_stacktrace=1 ENV PYTEST_ADDOPTS --dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_postgresql_database_engine/test.py::test_datetime test_postgresql_database_engine/test.py::test_get_create_table_query_with_multidim_arrays test_postgresql_database_engine/test.py::test_inaccessible_postgresql_database_engine_filterable_on_system_tables test_postgresql_database_engine/test.py::test_postgres_database_engine_with_postgres_ddl test_postgresql_database_engine/test.py::test_postgres_database_old_syntax test_postgresql_database_engine/test.py::test_postgresql_database_engine_queries test_postgresql_database_engine/test.py::test_postgresql_database_engine_table_cache test_postgresql_database_engine/test.py::test_postgresql_database_engine_with_clickhouse_ddl test_postgresql_database_engine/test.py::test_postgresql_database_with_schema test_postgresql_database_engine/test.py::test_postgresql_fetch_tables test_postgresql_database_engine/test.py::test_postgresql_password_leak test_postgresql_database_engine/test.py::test_predefined_connection_configuration test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order test_prometheus_endpoint/test.py::test_prometheus_endpoint test_prometheus_protocols/test.py::test_64bit_id test_prometheus_protocols/test.py::test_create_as_table test_prometheus_protocols/test.py::test_custom_id_algorithm test_prometheus_protocols/test.py::test_default test_prometheus_protocols/test.py::test_external_tables test_prometheus_protocols/test.py::test_inner_engines test_prometheus_protocols/test.py::test_read_auth test_prometheus_protocols/test.py::test_remote_write_v1_status_code test_prometheus_protocols/test.py::test_tags_to_columns test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_read_only_table/test.py::test_restart_zookeeper test_recompression_ttl/test.py::test_recompression_multiple_ttls test_recompression_ttl/test.py::test_recompression_replicated test_recompression_ttl/test.py::test_recompression_simple test_recovery_time_metric/test.py::test_recovery_time_metric test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db test_refreshable_mv/test.py::test_refreshable_mv_in_system_db test_relative_filepath/test.py::test_filepath test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers test_reload_certificate/test.py::test_ECcert_reload test_reload_certificate/test.py::test_cert_with_pass_phrase test_reload_certificate/test.py::test_chain_reload test_reload_certificate/test.py::test_first_than_second_cert test_reload_clusters_config/test.py::test_add_cluster test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_simple_reload test_reload_clusters_config/test.py::test_update_one_cluster test_reloading_settings_from_users_xml/test.py::test_force_reload test_reloading_settings_from_users_xml/test.py::test_reload_on_timeout test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_enum test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_int test_reloading_settings_from_users_xml/test.py::test_unknown_setting_force_reload test_reloading_settings_from_users_xml/test.py::test_unknown_setting_reload_on_timeout 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain]' test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4]' test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format test_render_log_file_name_templates/test.py::test_check_file_names test_replica_can_become_leader/test.py::test_can_become_leader test_replica_is_active/test.py::test_replica_is_active test_replicated_zero_copy_projection_mutation/test.py::test_all_projection_files_are_dropped_when_part_is_dropped test_replicated_zero_copy_projection_mutation/test.py::test_hardlinks_preserved_when_projection_dropped test_replicating_constants/test.py::test_different_versions test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_restart_server/test.py::test_drop_memory_database test_restart_server/test.py::test_flushes_async_insert_queue test_restore_replica/test.py::test_restore_replica_alive_replicas test_restore_replica/test.py::test_restore_replica_invalid_tables test_restore_replica/test.py::test_restore_replica_parallel test_restore_replica/test.py::test_restore_replica_sequential test_rocksdb_read_only/test.py::test_dirctory_missing_after_stop test_rocksdb_read_only/test.py::test_read_only test_role/test.py::test_admin_option test_role/test.py::test_changing_default_roles_affects_new_sessions_only test_role/test.py::test_combine_privileges test_role/test.py::test_create_role test_role/test.py::test_function_current_roles test_role/test.py::test_grant_role_to_role test_role/test.py::test_introspection test_role/test.py::test_revoke_requires_admin_option 'test_role/test.py::test_role_expiration[False]' 'test_role/test.py::test_role_expiration[True]' test_role/test.py::test_roles_cache test_role/test.py::test_set_role test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable 'test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header]' test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated -vvv -ss Cluster start called. is_up=False ENV CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge ENV COMPOSE_HTTP_TIMEOUT 600 ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ENV DOCKER_KERBERIZED_HADOOP_TAG latest ENV DOCKER_CHANNEL stable ENV DOCKER_CLIENT_TIMEOUT 300 Stdout:net.ipv4.ip_local_port_range = 55000 65535 ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e ENV PWD /ClickHouse/tests/integration Running tests in /ClickHouse/tests/integration/test_reload_certificate/test.py ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 ENV CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge Cluster start called. is_up=False ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config ENV TZ Etc/UTC ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java ENV DOCKER_BASE_TAG 8b2301119731 ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 ENV LC_CTYPE C.UTF-8 ENV INTEGRATION_TESTS_RUN_ID 0 ENV WORKER_FREE_PORTS 30050 30051 30052 30053 30054 30055 30056 30057 30058 30059 30060 30061 30062 30063 30064 30065 30066 30067 30068 30069 30070 30071 30072 30073 30074 30075 30076 30077 30078 30079 30080 30081 30082 30083 30084 30085 30086 30087 30088 30089 30090 30091 30092 30093 30094 30095 30096 30097 30098 30099 ENV PYTEST_XDIST_TESTRUNUID 63960e6506124119bf68bf57c6473af0 ENV PYTEST_XDIST_WORKER gw1 ENV PYTEST_XDIST_WORKER_COUNT 10 ENV PYTEST_CURRENT_TEST test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] (setup) CLUSTER INIT base_config_dir:/clickhouse-config clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Setup Keeper Cluster name:backward_compatibility project_name:roottestremoteblobsnamingbackwardcompatibility-gw1. Added instance name:node tag:8b2301119731 base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/.env', '--project-name', 'roottestremoteblobsnamingbackwardcompatibility-gw1', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Cluster name:backward_compatibility project_name:roottestremoteblobsnamingbackwardcompatibility-gw1. Added instance name:new_node tag:8b2301119731 base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/.env', '--project-name', 'roottestremoteblobsnamingbackwardcompatibility-gw1', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/new_node/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Cluster name:backward_compatibility project_name:roottestremoteblobsnamingbackwardcompatibility-gw1. Added instance name:switching_node tag:8b2301119731 base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/.env', '--project-name', 'roottestremoteblobsnamingbackwardcompatibility-gw1', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/new_node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/switching_node/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ Starting cluster... Running tests in /ClickHouse/tests/integration/test_remote_blobs_naming/test_backward_compatibility.py Cluster start called. is_up=False Docker networks for project roottestreplicationcredentials-gw8 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestprometheusprotocols-gw3 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestrestorereplica-gw9 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestpostgresqldatabaseengine-gw2 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestreloadclustersconfig-gw7 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestreloadingsettingsfromusersxml-gw5 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottests3cluster-gw4 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestremoteblobsnamingbackwardcompatibility-gw1 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestreloadcertificate-gw6 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestprometheusprotocols-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestreplicationcredentials-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestrestorereplica-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestpostgresqldatabaseengine-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestreloadclustersconfig-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestreloadingsettingsfromusersxml-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottests3cluster-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestremoteblobsnamingbackwardcompatibility-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestreloadcertificate-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestprometheusprotocols-gw3 are DRIVER VOLUME NAME Cleanup called Docker volumes for project roottestreplicationcredentials-gw8 are DRIVER VOLUME NAME Docker volumes for project roottestreloadclustersconfig-gw7 are DRIVER VOLUME NAME Cleanup called Cleanup called Docker volumes for project roottestpostgresqldatabaseengine-gw2 are DRIVER VOLUME NAME Cleanup called Docker volumes for project roottestreloadingsettingsfromusersxml-gw5 are DRIVER VOLUME NAME Docker volumes for project roottestrestorereplica-gw9 are DRIVER VOLUME NAME Cleanup called Cleanup called Docker volumes for project roottestremoteblobsnamingbackwardcompatibility-gw1 are DRIVER VOLUME NAME Cleanup called Docker volumes for project roottests3cluster-gw4 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestreplicationcredentials-gw8 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestpostgresqldatabaseengine-gw2 are NETWORK ID NAME DRIVER SCOPE Docker volumes for project roottestreloadcertificate-gw6 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestrestorereplica-gw9 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestreloadclustersconfig-gw7 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestremoteblobsnamingbackwardcompatibility-gw1 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestprometheusprotocols-gw3 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottests3cluster-gw4 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestreloadingsettingsfromusersxml-gw5 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicationcredentials-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestpostgresqldatabaseengine-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestrestorereplica-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestprometheusprotocols-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestreloadclustersconfig-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestreloadingsettingsfromusersxml-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestremoteblobsnamingbackwardcompatibility-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker networks for project roottestreloadcertificate-gw6 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottests3cluster-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreplicationcredentials-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicationcredentials-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] Docker volumes for project roottestprometheusprotocols-gw3 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestprometheusprotocols-gw3-.*-1$' --format '{{.ID}}:{{.Names}}'] Docker volumes for project roottestpostgresqldatabaseengine-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestpostgresqldatabaseengine-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Docker volumes for project roottestrestorereplica-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrestorereplica-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Docker volumes for project roottestreloadclustersconfig-gw7 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreloadclustersconfig-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] Docker containers for project roottestreloadcertificate-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestremoteblobsnamingbackwardcompatibility-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestremoteblobsnamingbackwardcompatibility-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Docker volumes for project roottests3cluster-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottests3cluster-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreplicationcredentials-gw8 Trying to prune unused networks... Docker volumes for project roottestreloadingsettingsfromusersxml-gw5 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreloadingsettingsfromusersxml-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreloadclustersconfig-gw7 Trying to prune unused networks... Unstopped containers: {} No running containers for project: roottestpostgresqldatabaseengine-gw2 Trying to prune unused networks... Unstopped containers: {} No running containers for project: roottestrestorereplica-gw9 Trying to prune unused networks... Docker volumes for project roottestreloadcertificate-gw6 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreloadcertificate-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestprometheusprotocols-gw3 Trying to prune unused networks... Unstopped containers: {} No running containers for project: roottestremoteblobsnamingbackwardcompatibility-gw1 Trying to prune unused networks... Unstopped containers: {} No running containers for project: roottestreloadingsettingsfromusersxml-gw5 Trying to prune unused networks... Unstopped containers: {} No running containers for project: roottests3cluster-gw4 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Trying to prune unused images... Unstopped containers: {} Command:[docker image prune -f] No running containers for project: roottestreloadcertificate-gw6 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Trying to prune unused images... Command:[docker image prune -f] Trying to prune unused images... Command:[docker image prune -f] Trying to prune unused images... Command:[docker image prune -f] Trying to prune unused images... Command:[docker image prune -f] Trying to prune unused images... Command:[docker image prune -f] Stderr:Error response from daemon: a prune operation is already running Exitcode:1 Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Trying to prune unused images... Command:[docker image prune -f] Stderr:Error response from daemon: a prune operation is already running Exitcode:1 Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stderr:Error response from daemon: a prune operation is already running Exitcode:1 Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stderr:Error response from daemon: a prune operation is already running Exitcode:1 Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_replication_credentials/configs/remote_servers.xml', '/ClickHouse/tests/integration/test_replication_credentials/configs/credentials1.xml'] to /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/database Setup logs dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Stdout:1 Volumes pruned: 1 Create directory for configuration generated in this helper Setup directory for instance: node Create directory for common tests configuration Copy common configuration from helpers Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Stdout:Total reclaimed space: 0B Generate and write macros file Generate and write macros file Images pruned Copy custom test config files ['/ClickHouse/tests/integration/test_replication_credentials/configs/remote_servers.xml', '/ClickHouse/tests/integration/test_replication_credentials/configs/credentials1.xml'] to /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/configs/config.d Trying to prune unused volumes... Copy custom test config files ['/ClickHouse/tests/integration/test_reload_clusters_config/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/node/configs/config.d Command:[docker volume ls | wc -l] Setup database dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/database Setup logs dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup database dir /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/node/database Setup directory for instance: node3 Setup logs dir /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/node/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Create directory for configuration generated in this helper Create directory for common tests configuration Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/.env Stdout:1 Copy common configuration from helpers Volumes pruned: 1 Setup directory for instance: node Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_replication_credentials/configs/remote_servers.xml', '/ClickHouse/tests/integration/test_replication_credentials/configs/no_credentials.xml'] to /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/configs/config.d Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Create directory for configuration generated in this helper Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Setup database dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/database No config file found Create directory for common tests configuration Setup logs dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/logs Copy common configuration from helpers Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node4 Generate and write macros file Stdout:1 Create directory for configuration generated in this helper Copy custom test config files ['/ClickHouse/tests/integration/test_prometheus_protocols/configs/prometheus.xml'] to /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/node/configs/config.d Volumes pruned: 1 Create directory for common tests configuration Setup directory for instance: node Copy common configuration from helpers Create directory for configuration generated in this helper Create directory for common tests configuration Generate and write macros file Copy common configuration from helpers Setup database dir /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/node/database Setup logs dir /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/node/logs Copy custom test config files ['/ClickHouse/tests/integration/test_replication_credentials/configs/remote_servers.xml', '/ClickHouse/tests/integration/test_replication_credentials/configs/no_credentials.xml'] to /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/configs/config.d Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'PROMETHEUS_WRITER_HOST': 'prometheus_writer', 'PROMETHEUS_WRITER_PORT': '9090', 'PROMETHEUS_WRITER_LOGS': '/ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/prometheus_writer/logs', 'PROMETHEUS_WRITER_LOGS_FS': 'bind', 'PROMETHEUS_READER_HOST': 'prometheus_reader', 'PROMETHEUS_READER_PORT': '9091', 'PROMETHEUS_READER_LOGS': '/ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/prometheus_reader/logs', 'PROMETHEUS_READER_LOGS_FS': 'bind', 'PROMETHEUS_REMOTE_WRITE_HANDLER': 'http://node:9092/write', 'PROMETHEUS_REMOTE_READ_HANDLER': 'http://node:9092/read'} stored in /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/.env Setup database dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/database Generate and write macros file Setup logs dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Stdout:1 Setup directory for instance: node5 Volumes pruned: 1 Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Setup directory for instance: node1 Copy custom test config files ['/ClickHouse/tests/integration/test_remote_blobs_naming/configs/old_node.xml', '/ClickHouse/tests/integration/test_remote_blobs_naming/configs/storage_conf.xml'] to /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/node/configs/config.d No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Create directory for configuration generated in this helper No config file found Create directory for common tests configuration Copy common configuration from helpers Create directory for configuration generated in this helper Create directory for common tests configuration Generate and write macros file Copy common configuration from helpers Copy custom test config files ['/ClickHouse/tests/integration/test_replication_credentials/configs/remote_servers.xml', '/ClickHouse/tests/integration/test_replication_credentials/configs/credentials1.xml'] to /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/configs/config.d Setup database dir /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/node/database Setup logs dir /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/node/logs Generate and write macros file Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup database dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/database Copy custom test config files ['/ClickHouse/tests/integration/test_postgresql_database_engine/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/node1/configs/config.d Setup directory for instance: new_node Setup logs dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node6 Create directory for configuration generated in this helper Setup database dir /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/node1/database Create directory for common tests configuration Setup logs dir /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/node1/logs Copy common configuration from helpers Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'POSTGRES_PORT': '5432', 'POSTGRES_DIR': '/ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/postgres/postgres1', 'POSTGRES_LOGS_FS': 'bind'} stored in /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/.env Create directory for configuration generated in this helper Create directory for common tests configuration Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Generate and write macros file No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Copy common configuration from helpers No config file found Copy custom test config files ['/ClickHouse/tests/integration/test_remote_blobs_naming/configs/new_node.xml', '/ClickHouse/tests/integration/test_remote_blobs_naming/configs/storage_conf_new.xml'] to /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/new_node/configs/config.d Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_replication_credentials/configs/remote_servers.xml', '/ClickHouse/tests/integration/test_replication_credentials/configs/credentials2.xml'] to /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/configs/config.d Setup database dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/database Setup database dir /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/new_node/database Setup logs dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node7 Setup logs dir /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/new_node/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Create directory for configuration generated in this helper Setup directory for instance: switching_node Create directory for common tests configuration Copy common configuration from helpers Stdout:1 Create directory for configuration generated in this helper Volumes pruned: 1 Generate and write macros file Setup directory for instance: s0_0_0 Create directory for common tests configuration Copy common configuration from helpers Stdout:1 Copy custom test config files ['/ClickHouse/tests/integration/test_replication_credentials/configs/remote_servers.xml', '/ClickHouse/tests/integration/test_replication_credentials/configs/credentials1.xml'] to /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/configs/config.d Volumes pruned: 1 Setup directory for instance: replica1 Setup database dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/database Setup logs dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/logs Generate and write macros file Create directory for configuration generated in this helper Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Create directory for common tests configuration Copy custom test config files ['/ClickHouse/tests/integration/test_remote_blobs_naming/configs/switching_node.xml', '/ClickHouse/tests/integration/test_remote_blobs_naming/configs/storage_conf.xml'] to /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/switching_node/configs/config.d Create directory for configuration generated in this helper Setup directory for instance: node8 Copy common configuration from helpers Create directory for common tests configuration Copy common configuration from helpers Create directory for configuration generated in this helper Create directory for common tests configuration Generate and write macros file http://localhost:None "GET /version HTTP/1.1" 200 826 Copy common configuration from helpers Generate and write macros file Setup database dir /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/switching_node/database Copy custom test config files ['/ClickHouse/tests/integration/test_restore_replica/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica1/configs/config.d Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_0/configs/config.d Generate and write macros file Setup logs dir /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/switching_node/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/.env --project-name roottestreloadclustersconfig-gw7 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml pull] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper3/coordination', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/minio/certs/public.crt', 'RESOLVER_LOGS': '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/resolver', 'RESOLVER_LOGS_FS': 'bind'} stored in /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/.env Copy custom test config files ['/ClickHouse/tests/integration/test_replication_credentials/configs/remote_servers.xml', '/ClickHouse/tests/integration/test_replication_credentials/configs/no_credentials.xml'] to /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/configs/config.d Setup database dir /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica1/database Setup logs dir /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica1/logs Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup database dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/database Setup logs dir /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/logs No config file found Setup directory for instance: replica2 Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Stdout:1 No config file found Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/.env Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_0/database Volumes pruned: 1 Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_0/logs http://localhost:None "GET /version HTTP/1.1" 200 826 Setup directory for instance: node Create directory for configuration generated in this helper Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Create directory for common tests configuration Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Setup directory for instance: s0_0_1 No config file found Copy common configuration from helpers Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Command:[docker compose --env-file /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/.env --project-name roottestprometheusprotocols-gw3 --file /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_prometheus.yml pull] No config file found Create directory for configuration generated in this helper Create directory for configuration generated in this helper Generate and write macros file Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Copy custom test config files ['/ClickHouse/tests/integration/test_restore_replica/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica2/database Setup logs dir /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Generate and write macros file Setup directory for instance: replica3 Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_1/configs/config.d Create directory for configuration generated in this helper Copy custom test config files [] to /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/_instances-0-gw5/node/configs/config.d Create directory for common tests configuration Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_1/database Copy common configuration from helpers Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: s0_1_0 Generate and write macros file Setup database dir /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/_instances-0-gw5/node/database Create directory for configuration generated in this helper Copy custom test config files ['/ClickHouse/tests/integration/test_restore_replica/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica3/configs/config.d Setup logs dir /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/_instances-0-gw5/node/logs Create directory for common tests configuration Setup database dir /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica3/database Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Copy common configuration from helpers Setup logs dir /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica3/logs Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/_instances-0-gw5/.env Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Generate and write macros file No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Stdout:1 Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_1_0/configs/config.d No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Volumes pruned: 1 No config file found Setup directory for instance: node Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_1_0/database Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_1_0/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper3/coordination', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/minio/certs/public.crt', 'RESOLVER_LOGS': '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/resolver', 'RESOLVER_LOGS_FS': 'bind'} stored in /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/.env Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_reload_certificate/configs/first.crt', '/ClickHouse/tests/integration/test_reload_certificate/configs/first.key', '/ClickHouse/tests/integration/test_reload_certificate/configs/second.crt', '/ClickHouse/tests/integration/test_reload_certificate/configs/second.key', '/ClickHouse/tests/integration/test_reload_certificate/configs/ECcert.crt', '/ClickHouse/tests/integration/test_reload_certificate/configs/ECcert.key', '/ClickHouse/tests/integration/test_reload_certificate/configs/WithChain.crt', '/ClickHouse/tests/integration/test_reload_certificate/configs/WithChain.key', '/ClickHouse/tests/integration/test_reload_certificate/configs/WithPassPhrase.crt', '/ClickHouse/tests/integration/test_reload_certificate/configs/WithPassPhrase.key', '/ClickHouse/tests/integration/test_reload_certificate/configs/cert.xml'] to /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/node/configs/config.d http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/.env --project-name roottestpostgresqldatabaseengine-gw2 --file /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_postgres.yml pull] Setup database dir /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/node/database Setup logs dir /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/node/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 http://localhost:None "GET /version HTTP/1.1" 200 826 http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/_instances-0-gw5/.env --project-name roottestreloadingsettingsfromusersxml-gw5 --file /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/_instances-0-gw5/node/docker-compose.yml pull] Command:[docker compose --env-file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/.env --project-name roottestreplicationcredentials-gw8 --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/docker-compose.yml pull] Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/.env --project-name roottests3cluster-gw4 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_1_0/docker-compose.yml pull] http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/.env --project-name roottestremoteblobsnamingbackwardcompatibility-gw1 --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/new_node/docker-compose.yml --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/switching_node/docker-compose.yml pull] http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/.env --project-name roottestrestorereplica-gw9 --file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica2/docker-compose.yml --file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica3/docker-compose.yml pull] http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/.env --project-name roottestreloadcertificate-gw6 --file /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/node/docker-compose.yml pull] Stdout:Deleted Networks: Stdout:roottestmysqlprotocol-gw0_default Stdout: Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Stdout:net.ipv4.ip_local_port_range = 55000 65535 Running tests in /ClickHouse/tests/integration/test_role/test.py Cluster start called. is_up=False Docker networks for project roottestrole-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrole-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrole-gw0 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestrole-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrole-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrole-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrole-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestrole-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: instance Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_role/_instances-0-gw0/instance/configs/config.d Setup database dir /ClickHouse/tests/integration/test_role/_instances-0-gw0/instance/database Setup logs dir /ClickHouse/tests/integration/test_role/_instances-0-gw0/instance/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_role/_instances-0-gw0/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_role/_instances-0-gw0/.env --project-name roottestrole-gw0 --file /ClickHouse/tests/integration/test_role/_instances-0-gw0/instance/docker-compose.yml pull] Stderr: instance Pulling Stderr: instance Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_role/_instances-0-gw0/.env --project-name roottestrole-gw0 --file /ClickHouse/tests/integration/test_role/_instances-0-gw0/instance/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_role/_instances-0-gw0/.env --project-name roottestrole-gw0 --file /ClickHouse/tests/integration/test_role/_instances-0-gw0/instance/docker-compose.yml up -d --no-recreate] Stderr: node Pulling Stderr: node Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/.env --project-name roottestreloadcertificate-gw6 --file /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/.env --project-name roottestreloadcertificate-gw6 --file /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/node/docker-compose.yml up -d --no-recreate] Stderr: postgres1 Pulling Stderr: node1 Pulling Stderr: node1 Pulled Stderr: postgres1 Pulled Setup Postgres Command:[docker compose --project-name roottestpostgresqldatabaseengine-gw2 --env-file /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_postgres.yml --verbose up -d] Stderr: zoo1 Skipped - Image is already being pulled by zoo3 Stderr: zoo3 Pulling Stderr: node8 Skipped - Image is already being pulled by node5 Stderr: zoo2 Skipped - Image is already being pulled by zoo3 Stderr: node7 Skipped - Image is already being pulled by node5 Stderr: replica2 Skipped - Image is already being pulled by zoo3 Stderr: zoo2 Skipped - Image is already being pulled by node5 Stderr: replica3 Skipped - Image is already being pulled by zoo3 Stderr: zoo3 Skipped - Image is already being pulled by node5 Stderr: node4 Skipped - Image is already being pulled by node5 Stderr: node3 Skipped - Image is already being pulled by node5 Stderr: replica1 Skipped - Image is already being pulled by zoo3 Stderr: node1 Skipped - Image is already being pulled by node5 Stderr: node6 Skipped - Image is already being pulled by node5 Stderr: zoo3 Pulled Stderr: zoo1 Skipped - Image is already being pulled by node5 Stderr: node2 Skipped - Image is already being pulled by node5 Setup ZooKeeper Stderr: node5 Pulling Stderr: node5 Pulled Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper1/log', '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper1/config', '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper1/coordination', '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper2/log', '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper2/config', '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper2/coordination', '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper3/log', '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper3/config', '/ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/keeper3/coordination'] Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper1/log', '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper1/config', '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper1/coordination', '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper2/log', '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper2/config', '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper2/coordination', '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper3/log', '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper3/config', '/ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/keeper3/coordination'] Command:[docker compose --project-name roottestrestorereplica-gw9 --env-file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Command:[docker compose --project-name roottestreplicationcredentials-gw8 --env-file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr: node Pulling Stderr: node Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/_instances-0-gw5/.env --project-name roottestreloadingsettingsfromusersxml-gw5 --file /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/_instances-0-gw5/node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/_instances-0-gw5/.env --project-name roottestreloadingsettingsfromusersxml-gw5 --file /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/_instances-0-gw5/node/docker-compose.yml up -d --no-recreate] Stderr: new_node Skipped - Image is already being pulled by node Stderr: zoo1 Skipped - Image is already being pulled by node Stderr: switching_node Skipped - Image is already being pulled by node Stderr: zoo3 Skipped - Image is already being pulled by node Stderr: proxy2 Skipped - Image is already being pulled by proxy1 Stderr: zoo2 Skipped - Image is already being pulled by node Stderr: minio1 Pulling Stderr: proxy1 Pulling Stderr: resolver Pulling Stderr: node Pulling Stderr: s0_1_0 Skipped - Image is already being pulled by zoo1 Stderr: resolver Pulled Stderr: zoo3 Skipped - Image is already being pulled by zoo1 Stderr: s0_0_1 Skipped - Image is already being pulled by zoo1 Stderr: node Pulled Stderr: zoo2 Skipped - Image is already being pulled by zoo1 Stderr: minio1 Pulled Stderr: proxy2 Skipped - Image is already being pulled by proxy1 Stderr: s0_0_0 Skipped - Image is already being pulled by zoo1 Stderr: proxy1 Pulled Stderr: resolver Pulling Stderr: proxy1 Pulling Setup ZooKeeper Stderr: zoo1 Pulling Stderr: minio1 Pulling Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper1/log', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper1/config', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper1/coordination', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper2/log', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper2/config', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper2/coordination', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper3/log', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper3/config', '/ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/keeper3/coordination'] Stderr: zoo1 Pulled Stderr: minio1 Pulled Stderr: proxy1 Pulled Stderr: resolver Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper1/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper1/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper1/coordination', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper2/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper2/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper2/coordination', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper3/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper3/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/keeper3/coordination'] Command:[docker compose --project-name roottestremoteblobsnamingbackwardcompatibility-gw1 --env-file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Command:[docker compose --project-name roottests3cluster-gw4 --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr: zoo2 Skipped - Image is already being pulled by zoo1 Stderr: node Skipped - Image is already being pulled by zoo1 Stderr: zoo3 Skipped - Image is already being pulled by zoo1 Stderr: zoo1 Pulling Stderr: zoo1 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper1/log', '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper1/config', '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper1/coordination', '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper2/log', '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper2/config', '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper2/coordination', '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper3/log', '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper3/config', '/ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/keeper3/coordination'] Command:[docker compose --project-name roottestreloadclustersconfig-gw7 --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr: Network roottestrole-gw0_default Creating Stderr: Network roottestrole-gw0_default Created Stderr: Container roottestrole-gw0-instance-1 Creating Stderr: Container roottestrole-gw0-instance-1 Created Stderr: Container roottestrole-gw0-instance-1 Starting Stderr: Container roottestrole-gw0-instance-1 Started ClickHouse instance created get_instance_ip instance_name=instance http://localhost:None "GET /v1.46/containers/roottestrole-gw0-instance-1/json HTTP/1.1" 200 None get_instance_ip instance_name=instance http://localhost:None "GET /v1.46/containers/roottestrole-gw0-instance-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in instance, ip: 172.16.1.2... http://localhost:None "GET /v1.46/containers/roottestrole-gw0-instance-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/09f5b8e3aab024c12a268058aa930baef89792c9a71e0a14d7978e6f05912a51/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/09f5b8e3aab024c12a268058aa930baef89792c9a71e0a14d7978e6f05912a51/json HTTP/1.1" 200 None Stderr: Network roottestreloadcertificate-gw6_default Creating Stderr: Network roottestreloadcertificate-gw6_default Created Stderr: Container roottestreloadcertificate-gw6-node-1 Creating Stderr: Container roottestreloadcertificate-gw6-node-1 Created Stderr: Container roottestreloadcertificate-gw6-node-1 Starting Stderr: Container roottestreloadcertificate-gw6-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestreloadcertificate-gw6-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestreloadcertificate-gw6-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.2.2... http://localhost:None "GET /v1.46/containers/roottestreloadcertificate-gw6-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e63113fc1cc1e7b50ba7614a6903d9070bd829ccefd29aba0f141623e938969a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/09f5b8e3aab024c12a268058aa930baef89792c9a71e0a14d7978e6f05912a51/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e63113fc1cc1e7b50ba7614a6903d9070bd829ccefd29aba0f141623e938969a/json HTTP/1.1" 200 None Stderr:time="2025-04-02T03:58:17Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestpostgresqldatabaseengine-gw2_default Creating Stderr: Network roottestpostgresqldatabaseengine-gw2_default Created Stderr: Container roottestpostgresqldatabaseengine-gw2-postgres1-1 Creating Stderr: Container roottestpostgresqldatabaseengine-gw2-postgres1-1 Created Stderr: Container roottestpostgresqldatabaseengine-gw2-postgres1-1 Starting Stderr: Container roottestpostgresqldatabaseengine-gw2-postgres1-1 Started Stderr:time="2025-04-02T03:58:17Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:58:17Z" level=debug msg="otel error" error="" get_instance_ip instance_name=postgres1 http://localhost:None "GET /v1.46/containers/roottestpostgresqldatabaseengine-gw2-postgres1-1/json HTTP/1.1" 200 None Can't connect to Postgres connection to server at "172.16.3.2", port 5432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections? http://localhost:None "GET /v1.46/containers/09f5b8e3aab024c12a268058aa930baef89792c9a71e0a14d7978e6f05912a51/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e63113fc1cc1e7b50ba7614a6903d9070bd829ccefd29aba0f141623e938969a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/09f5b8e3aab024c12a268058aa930baef89792c9a71e0a14d7978e6f05912a51/json HTTP/1.1" 200 None ClickHouse instance started Executing query CREATE TABLE test_table(x UInt32, y UInt32) ENGINE = MergeTree ORDER BY tuple() on instance http://localhost:None "GET /v1.46/containers/e63113fc1cc1e7b50ba7614a6903d9070bd829ccefd29aba0f141623e938969a/json HTTP/1.1" 200 None Stderr:time="2025-04-02T03:58:17Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottests3cluster-gw4_default Creating Stderr: Network roottests3cluster-gw4_default Created Stderr: Container roottests3cluster-gw4-zoo3-1 Creating Stderr: Container roottests3cluster-gw4-zoo2-1 Creating Stderr: Container roottests3cluster-gw4-zoo1-1 Creating Stderr: Container roottests3cluster-gw4-zoo1-1 Created Stderr: Container roottests3cluster-gw4-zoo2-1 Created Stderr: Container roottests3cluster-gw4-zoo3-1 Created Stderr: Container roottests3cluster-gw4-zoo1-1 Starting Stderr: Container roottests3cluster-gw4-zoo2-1 Starting Stderr: Container roottests3cluster-gw4-zoo3-1 Starting Stderr: Container roottests3cluster-gw4-zoo3-1 Started Stderr: Container roottests3cluster-gw4-zoo2-1 Started Stderr: Container roottests3cluster-gw4-zoo1-1 Started Stderr:time="2025-04-02T03:58:18Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:58:18Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw4-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.4.3, port:2181, use_ssl:False Connecting to 172.16.4.3(172.16.4.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query INSERT INTO test_table VALUES (1,5), (2,10) on instance http://localhost:None "GET /v1.46/containers/e63113fc1cc1e7b50ba7614a6903d9070bd829ccefd29aba0f141623e938969a/json HTTP/1.1" 200 None ClickHouse node started run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cat > /etc/clickhouse-server/config.d/cert.xml << EOF\n\n 8443\n \n \n /etc/clickhouse-server/config.d/first.crt\n /etc/clickhouse-server/config.d/first.key\n true\n true\n sslv2,sslv3\n true\n \n \n \n\nEOF'] Command:[docker exec roottestreloadcertificate-gw6-node-1 bash -c cat > /etc/clickhouse-server/config.d/cert.xml << EOF 8443 /etc/clickhouse-server/config.d/first.crt /etc/clickhouse-server/config.d/first.key true true sslv2,sslv3 true EOF] Executing query SYSTEM RELOAD CONFIG on node Executing query CREATE USER A on instance Connecting to 172.16.4.3(172.16.4.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query CREATE USER B on instance run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/first.crt', 'https://localhost:8443/'] Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/first.crt https://localhost:8443/] Can't connect to Postgres connection to server at "172.16.3.2", port 5432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections? Stdout:Ok. run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/ECcert.crt', 'https://localhost:8443/'] Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/ECcert.crt https://localhost:8443/] Exitcode:60 run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cat > /etc/clickhouse-server/config.d/cert.xml << EOF\n\n 8443\n \n \n /etc/clickhouse-server/config.d/ECcert.crt\n /etc/clickhouse-server/config.d/ECcert.key\n true\n true\n sslv2,sslv3\n true\n \n \n \n\nEOF'] Command:[docker exec roottestreloadcertificate-gw6-node-1 bash -c cat > /etc/clickhouse-server/config.d/cert.xml << EOF 8443 /etc/clickhouse-server/config.d/ECcert.crt /etc/clickhouse-server/config.d/ECcert.key true true sslv2,sslv3 true EOF] Executing query SYSTEM RELOAD CONFIG on node Connecting to 172.16.4.3(172.16.4.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Stderr:time="2025-04-02T03:58:17Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestrestorereplica-gw9_default Creating Stderr: Network roottestrestorereplica-gw9_default Created Stderr: Container roottestrestorereplica-gw9-zoo1-1 Creating Stderr: Container roottestrestorereplica-gw9-zoo2-1 Creating Stderr: Container roottestrestorereplica-gw9-zoo3-1 Creating Stderr: Container roottestrestorereplica-gw9-zoo1-1 Created Stderr: Container roottestrestorereplica-gw9-zoo3-1 Created Stderr: Container roottestrestorereplica-gw9-zoo2-1 Created Stderr: Container roottestrestorereplica-gw9-zoo3-1 Starting Stderr: Container roottestrestorereplica-gw9-zoo1-1 Starting Stderr: Container roottestrestorereplica-gw9-zoo2-1 Starting Stderr: Container roottestrestorereplica-gw9-zoo3-1 Started Stderr: Container roottestrestorereplica-gw9-zoo1-1 Started Stderr: Container roottestrestorereplica-gw9-zoo2-1 Started Stderr:time="2025-04-02T03:58:18Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:58:18Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestrestorereplica-gw9-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.5.3, port:2181, use_ssl:False Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Stderr: Network roottestreloadingsettingsfromusersxml-gw5_default Creating Stderr: Network roottestreloadingsettingsfromusersxml-gw5_default Created Stderr: Container roottestreloadingsettingsfromusersxml-gw5-node-1 Creating Stderr: Container roottestreloadingsettingsfromusersxml-gw5-node-1 Created Stderr: Container roottestreloadingsettingsfromusersxml-gw5-node-1 Starting Stderr: Container roottestreloadingsettingsfromusersxml-gw5-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestreloadingsettingsfromusersxml-gw5-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestreloadingsettingsfromusersxml-gw5-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.6.2... http://localhost:None "GET /v1.46/containers/roottestreloadingsettingsfromusersxml-gw5-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c02a8df5520e68585c4cd44d0b3803fbb05ce9a9175c8f2b9de6060bfd07a5ca/json HTTP/1.1" 200 None run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/ECcert.crt', 'https://localhost:8443/'] Executing query CREATE ROLE R1 on instance Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/ECcert.crt https://localhost:8443/] Stdout:Ok. run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/first.crt', 'https://localhost:8443/'] Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/first.crt https://localhost:8443/] Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Executing query GRANT SELECT ON test_table TO R1 on instance Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/c02a8df5520e68585c4cd44d0b3803fbb05ce9a9175c8f2b9de6060bfd07a5ca/json HTTP/1.1" 200 None Exitcode:60 [gw6] PASSED test_reload_certificate/test.py::test_ECcert_reload test_reload_certificate/test.py::test_cert_with_pass_phrase run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cat > /etc/clickhouse-server/config.d/cert.xml << EOF\n\n 8443\n \n \n /etc/clickhouse-server/config.d/first.crt\n /etc/clickhouse-server/config.d/first.key\n true\n true\n sslv2,sslv3\n true\n \n \n \n\nEOF'] Command:[docker exec roottestreloadcertificate-gw6-node-1 bash -c cat > /etc/clickhouse-server/config.d/cert.xml << EOF 8443 /etc/clickhouse-server/config.d/first.crt /etc/clickhouse-server/config.d/first.key true true sslv2,sslv3 true EOF] Executing query SELECT * FROM test_table on instance Executing query SYSTEM RELOAD CONFIG on node http://localhost:None "GET /v1.46/containers/c02a8df5520e68585c4cd44d0b3803fbb05ce9a9175c8f2b9de6060bfd07a5ca/json HTTP/1.1" 200 None Executing query GRANT R1 TO A on instance Can't connect to Postgres connection to server at "172.16.3.2", port 5432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections? http://localhost:None "GET /v1.46/containers/c02a8df5520e68585c4cd44d0b3803fbb05ce9a9175c8f2b9de6060bfd07a5ca/json HTTP/1.1" 200 None Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/c02a8df5520e68585c4cd44d0b3803fbb05ce9a9175c8f2b9de6060bfd07a5ca/json HTTP/1.1" 200 None Stderr: prometheus_writer Skipped - Image is already being pulled by prometheus_reader Stderr: node Pulling Stderr: prometheus_reader Pulling Stderr: 9fa9226be034 Pulling fs layer Stderr: 1617e25568b2 Pulling fs layer Stderr: 52e274219e9a Pulling fs layer Stderr: 3d2f97fbf1fd Pulling fs layer Stderr: 4074b1353672 Pulling fs layer Stderr: 5425e01d7f3c Pulling fs layer Stderr: 0926657f3b6b Pulling fs layer Stderr: c9ecc1017088 Pulling fs layer Stderr: 238f9bf935c9 Pulling fs layer Stderr: 794f1dd56e5b Pulling fs layer Stderr: c8699fb3f236 Pulling fs layer Stderr: a9784cd47caf Pulling fs layer Stderr: 4074b1353672 Waiting Stderr: 0926657f3b6b Waiting Stderr: 5425e01d7f3c Waiting Stderr: c9ecc1017088 Waiting Stderr: 238f9bf935c9 Waiting Stderr: 794f1dd56e5b Waiting Stderr: c8699fb3f236 Waiting Stderr: 3d2f97fbf1fd Waiting Stderr: a9784cd47caf Waiting Stderr: node Pulled Stderr: 1617e25568b2 Downloading [=> ] 13.78kB/480.9kB Stderr: 9fa9226be034 Downloading [==> ] 35.5kB/783kB Stderr: 1617e25568b2 Downloading [==================================================>] 480.9kB/480.9kB Stderr: 9fa9226be034 Download complete Stderr: 9fa9226be034 Extracting [==> ] 32.77kB/783kB Stderr: 4074b1353672 Downloading [==================================================>] 604B/604B Stderr: 4074b1353672 Download complete Stderr: 9fa9226be034 Extracting [==================================================>] 783kB/783kB Stderr: 5425e01d7f3c Downloading [==================================================>] 2.677kB/2.677kB Stderr: 52e274219e9a Downloading [> ] 531.6kB/52.69MB Stderr: 0926657f3b6b Downloading [==================================================>] 3.088kB/3.088kB Stderr: 0926657f3b6b Verifying Checksum Stderr: 0926657f3b6b Download complete Stderr: c9ecc1017088 Downloading [=============================================> ] 3.647kB/4.022kB Stderr: c9ecc1017088 Downloading [==================================================>] 4.022kB/4.022kB Stderr: c9ecc1017088 Verifying Checksum Stderr: c9ecc1017088 Download complete Stderr: 238f9bf935c9 Downloading [==================================================>] 1.441kB/1.441kB Stderr: 238f9bf935c9 Verifying Checksum Stderr: 238f9bf935c9 Download complete Stderr: 3d2f97fbf1fd Downloading [> ] 503.8kB/47.38MB Stderr: 794f1dd56e5b Downloading [=> ] 3.645kB/138.8kB Stderr: 794f1dd56e5b Download complete Stderr: c8699fb3f236 Downloading [==================================================>] 100B/100B Stderr: c8699fb3f236 Verifying Checksum Stderr: c8699fb3f236 Download complete Stderr: a9784cd47caf Downloading [==================================================>] 723B/723B Stderr: a9784cd47caf Verifying Checksum Stderr: a9784cd47caf Download complete Stderr: 9fa9226be034 Pull complete Stderr: 1617e25568b2 Extracting [===> ] 32.77kB/480.9kB Stderr: 52e274219e9a Downloading [========================> ] 25.4MB/52.69MB Stderr: 3d2f97fbf1fd Downloading [==========================> ] 25.54MB/47.38MB Stderr: 52e274219e9a Downloading [=================================================> ] 52.03MB/52.69MB Stderr: 52e274219e9a Verifying Checksum Stderr: 1617e25568b2 Extracting [===============================================> ] 458.8kB/480.9kB Stderr: 52e274219e9a Download complete Stderr: 3d2f97fbf1fd Verifying Checksum Stderr: 3d2f97fbf1fd Download complete Stderr: 1617e25568b2 Extracting [==================================================>] 480.9kB/480.9kB Stderr: 1617e25568b2 Extracting [==================================================>] 480.9kB/480.9kB Stderr: 1617e25568b2 Pull complete Stderr: 52e274219e9a Extracting [> ] 557.1kB/52.69MB Stderr: 52e274219e9a Extracting [======> ] 7.242MB/52.69MB Stderr: 52e274219e9a Extracting [=============> ] 13.93MB/52.69MB Stderr: 52e274219e9a Extracting [===================> ] 20.05MB/52.69MB Stderr: 52e274219e9a Extracting [========================> ] 26.18MB/52.69MB Stderr: 52e274219e9a Extracting [========================================> ] 42.34MB/52.69MB Stderr: 52e274219e9a Extracting [================================================> ] 51.25MB/52.69MB Stderr: 52e274219e9a Extracting [==================================================>] 52.69MB/52.69MB Stderr: 52e274219e9a Pull complete Stderr: 3d2f97fbf1fd Extracting [> ] 491.5kB/47.38MB Stderr: 3d2f97fbf1fd Extracting [=======> ] 6.881MB/47.38MB Stderr: 3d2f97fbf1fd Extracting [============> ] 12.29MB/47.38MB Stderr: 3d2f97fbf1fd Extracting [=================> ] 16.71MB/47.38MB Stderr: 3d2f97fbf1fd Extracting [=====================> ] 20.15MB/47.38MB Stderr: 3d2f97fbf1fd Extracting [==============================> ] 29MB/47.38MB Stderr: 3d2f97fbf1fd Extracting [=================================================> ] 47.19MB/47.38MB Stderr: 3d2f97fbf1fd Extracting [==================================================>] 47.38MB/47.38MB Stderr: 3d2f97fbf1fd Pull complete Stderr: 4074b1353672 Extracting [==================================================>] 604B/604B Stderr: 4074b1353672 Extracting [==================================================>] 604B/604B Stderr: 4074b1353672 Pull complete Stderr: 5425e01d7f3c Extracting [==================================================>] 2.677kB/2.677kB Stderr: 5425e01d7f3c Extracting [==================================================>] 2.677kB/2.677kB Stderr: 5425e01d7f3c Pull complete Stderr: 0926657f3b6b Extracting [==================================================>] 3.088kB/3.088kB Stderr: 0926657f3b6b Extracting [==================================================>] 3.088kB/3.088kB Stderr: 0926657f3b6b Pull complete Stderr: c9ecc1017088 Extracting [==================================================>] 4.022kB/4.022kB Stderr: c9ecc1017088 Extracting [==================================================>] 4.022kB/4.022kB Stderr: c9ecc1017088 Pull complete Stderr: 238f9bf935c9 Extracting [==================================================>] 1.441kB/1.441kB Stderr: 238f9bf935c9 Extracting [==================================================>] 1.441kB/1.441kB Stderr: 238f9bf935c9 Pull complete Stderr: 794f1dd56e5b Extracting [===========> ] 32.77kB/138.8kB Stderr: 794f1dd56e5b Extracting [==================================================>] 138.8kB/138.8kB Stderr: 794f1dd56e5b Extracting [==================================================>] 138.8kB/138.8kB Stderr: 794f1dd56e5b Pull complete Stderr: c8699fb3f236 Extracting [==================================================>] 100B/100B Stderr: c8699fb3f236 Extracting [==================================================>] 100B/100B Stderr: c8699fb3f236 Pull complete Stderr: a9784cd47caf Extracting [==================================================>] 723B/723B Stderr: a9784cd47caf Extracting [==================================================>] 723B/723B Stderr: a9784cd47caf Pull complete Stderr: prometheus_reader Pulled Trying to create Prometheus instances by command docker compose --project-name roottestprometheusprotocols-gw3 --env-file /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_prometheus.yml --verbose up -d Command:[docker compose --project-name roottestprometheusprotocols-gw3 --env-file /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_prometheus.yml --verbose up -d] Connecting to 172.16.4.3(172.16.4.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/c02a8df5520e68585c4cd44d0b3803fbb05ce9a9175c8f2b9de6060bfd07a5ca/json HTTP/1.1" 200 None ClickHouse node started run container_id:roottestreloadingsettingsfromusersxml-gw5-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjEwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+Zmlyc3Rfb3JfcmFuZG9tPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4yPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml'] Command:[docker exec roottestreloadingsettingsfromusersxml-gw5-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjEwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+Zmlyc3Rfb3JfcmFuZG9tPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4yPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml] Executing query SYSTEM RELOAD CONFIG on node Executing query GRANT R1 TO B on instance run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/first.crt', 'https://localhost:8443/'] Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/first.crt https://localhost:8443/] Stderr:time="2025-04-02T03:58:17Z" level=trace msg="Docker Desktop integration not enabled" Stderr:time="2025-04-02T03:58:17Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestreplicationcredentials-gw8_default Creating Stderr: Network roottestreplicationcredentials-gw8_default Created Stderr: Container roottestreplicationcredentials-gw8-zoo3-1 Creating Stderr: Network roottestremoteblobsnamingbackwardcompatibility-gw1_default Creating Stderr: Container roottestreplicationcredentials-gw8-zoo1-1 Creating Stderr: Network roottestremoteblobsnamingbackwardcompatibility-gw1_default Created Stderr: Container roottestreplicationcredentials-gw8-zoo2-1 Creating Stderr: Container roottestreplicationcredentials-gw8-zoo3-1 Created Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo1-1 Creating Stderr: Container roottestreplicationcredentials-gw8-zoo1-1 Created Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo3-1 Creating Stderr: Container roottestreplicationcredentials-gw8-zoo2-1 Created Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo2-1 Creating Stderr: Container roottestreplicationcredentials-gw8-zoo2-1 Starting Stderr: Container roottestreplicationcredentials-gw8-zoo3-1 Starting Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo3-1 Created Stderr: Container roottestreplicationcredentials-gw8-zoo1-1 Starting Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo1-1 Created Stderr: Container roottestreplicationcredentials-gw8-zoo2-1 Started Stderr: Container roottestreplicationcredentials-gw8-zoo1-1 Started Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo2-1 Created Stderr: Container roottestreplicationcredentials-gw8-zoo3-1 Started Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo2-1 Starting Stderr:time="2025-04-02T03:58:19Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:58:19Z" level=debug msg="otel error" error="" Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo3-1 Starting Wait ZooKeeper to start Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo1-1 Starting get_instance_ip instance_name=zoo1 Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo2-1 Started Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo3-1 Started Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo1-1 Started Stderr:time="2025-04-02T03:58:19Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:58:19Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.7.3, port:2181, use_ssl:False http://localhost:None "GET /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-zoo1-1/json HTTP/1.1" 200 None Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused get_kazoo_client: zoo1, ip:172.16.8.4, port:2181, use_ssl:False Connecting to 172.16.8.4(172.16.8.4):2181, use_ssl: False Stdout:Ok. run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/WithPassPhrase.crt', 'https://localhost:8443/'] Connection dropped: socket connection error: Connection refused Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/WithPassPhrase.crt https://localhost:8443/] Exitcode:60 run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cat > /etc/clickhouse-server/config.d/cert.xml << EOF\n\n 8443\n \n \n /etc/clickhouse-server/config.d/WithPassPhrase.crt\n /etc/clickhouse-server/config.d/WithPassPhrase.key\n true\n true\n sslv2,sslv3\n true\n \n KeyFileHandler\n \n test\n \n \n\n \n \n\nEOF'] Command:[docker exec roottestreloadcertificate-gw6-node-1 bash -c cat > /etc/clickhouse-server/config.d/cert.xml << EOF 8443 /etc/clickhouse-server/config.d/WithPassPhrase.crt /etc/clickhouse-server/config.d/WithPassPhrase.key true true sslv2,sslv3 true KeyFileHandler test EOF] Executing query SYSTEM RELOAD CONFIG on node Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.8.4(172.16.8.4):2181, use_ssl: False Executing query SELECT * FROM test_table on instance Connection dropped: socket connection error: Connection refused Executing query SELECT getSetting('max_memory_usage') on node Postgres Started ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/.env --project-name roottestpostgresqldatabaseengine-gw2 --file /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_postgres.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/.env --project-name roottestpostgresqldatabaseengine-gw2 --file /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_postgres.yml up -d --no-recreate] Stderr:time="2025-04-02T03:58:17Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestreloadclustersconfig-gw7_default Creating Stderr: Network roottestreloadclustersconfig-gw7_default Created Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Creating Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Creating Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Creating Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Created Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Created Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Created Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Starting Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Starting Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Starting Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Started Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Started Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Started Stderr:time="2025-04-02T03:58:19Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:58:19Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestreloadclustersconfig-gw7-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.9.4, port:2181, use_ssl:False Connecting to 172.16.9.4(172.16.9.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/WithPassPhrase.crt', 'https://localhost:8443/'] Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/WithPassPhrase.crt https://localhost:8443/] Executing query SELECT getSetting('load_balancing') on node Executing query GRANT R1 TO A WITH ADMIN OPTION on instance Executing query GRANT R1 TO B on instance Connecting to 172.16.9.4(172.16.9.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Stdout:Ok. run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/first.crt', 'https://localhost:8443/'] Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/first.crt https://localhost:8443/] Executing query SELECT getSetting('alter_sync') on node Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Exitcode:60 [gw6] PASSED test_reload_certificate/test.py::test_cert_with_pass_phrase test_reload_certificate/test.py::test_chain_reload run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cat > /etc/clickhouse-server/config.d/cert.xml << EOF\n\n 8443\n \n \n /etc/clickhouse-server/config.d/first.crt\n /etc/clickhouse-server/config.d/first.key\n true\n true\n sslv2,sslv3\n true\n \n \n \n\nEOF'] Command:[docker exec roottestreloadcertificate-gw6-node-1 bash -c cat > /etc/clickhouse-server/config.d/cert.xml << EOF 8443 /etc/clickhouse-server/config.d/first.crt /etc/clickhouse-server/config.d/first.key true true sslv2,sslv3 true EOF] Executing query SELECT * FROM test_table on instance run container_id:roottestreloadingsettingsfromusersxml-gw5-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjIwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+bmVhcmVzdF9ob3N0bmFtZTwvbG9hZF9iYWxhbmNpbmc+CiAgICAgICAgICAgIDxyZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+MDwvcmVwbGljYXRpb25fYWx0ZXJfcGFydGl0aW9uc19zeW5jPgogICAgICAgIDwvZGVmYXVsdD4KICAgIDwvcHJvZmlsZXM+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/z.xml'] Command:[docker exec roottestreloadingsettingsfromusersxml-gw5-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjIwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+bmVhcmVzdF9ob3N0bmFtZTwvbG9hZF9iYWxhbmNpbmc+CiAgICAgICAgICAgIDxyZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+MDwvcmVwbGljYXRpb25fYWx0ZXJfcGFydGl0aW9uc19zeW5jPgogICAgICAgIDwvZGVmYXVsdD4KICAgIDwvcHJvZmlsZXM+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/z.xml] Executing query SYSTEM RELOAD CONFIG on node Connecting to 172.16.8.4(172.16.8.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query DROP USER IF EXISTS A, B on instance [gw0] PASSED test_role/test.py::test_admin_option Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SYSTEM RELOAD CONFIG on node run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/first.crt', 'https://localhost:8443/'] Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/first.crt https://localhost:8443/] Connecting to 172.16.9.4(172.16.9.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query DROP ROLE IF EXISTS R1, R2, R3, R4 on instance Stdout:Ok. run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/WithChain.crt', 'https://localhost:8443/'] Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/WithChain.crt https://localhost:8443/] Exitcode:60 run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cat > /etc/clickhouse-server/config.d/cert.xml << EOF\n\n 8443\n \n \n /etc/clickhouse-server/config.d/WithChain.crt\n /etc/clickhouse-server/config.d/WithChain.key\n true\n true\n sslv2,sslv3\n true\n \n \n \n\nEOF'] Command:[docker exec roottestreloadcertificate-gw6-node-1 bash -c cat > /etc/clickhouse-server/config.d/cert.xml << EOF 8443 /etc/clickhouse-server/config.d/WithChain.crt /etc/clickhouse-server/config.d/WithChain.key true true sslv2,sslv3 true EOF] Stderr: Container roottestpostgresqldatabaseengine-gw2-postgres1-1 Running Stderr: Container roottestpostgresqldatabaseengine-gw2-node1-1 Creating Stderr: Container roottestpostgresqldatabaseengine-gw2-node1-1 Created Stderr: Container roottestpostgresqldatabaseengine-gw2-node1-1 Starting Stderr: Container roottestpostgresqldatabaseengine-gw2-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestpostgresqldatabaseengine-gw2-node1-1/json HTTP/1.1" 200 None Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False get_instance_ip instance_name=node1 Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/roottestpostgresqldatabaseengine-gw2-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.3.3... http://localhost:None "GET /v1.46/containers/roottestpostgresqldatabaseengine-gw2-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5b774a794940bfbbca5c3fb9ddfab1b68bd1c868442bf1e0587fbeafc2c3ba01/json HTTP/1.1" 200 None Executing query SYSTEM RELOAD CONFIG on node Executing query SELECT getSetting('max_memory_usage') on node test_role/test.py::test_changing_default_roles_affects_new_sessions_only Executing query CREATE USER A on instance http://localhost:None "GET /v1.46/containers/5b774a794940bfbbca5c3fb9ddfab1b68bd1c868442bf1e0587fbeafc2c3ba01/json HTTP/1.1" 200 None Connecting to 172.16.9.4(172.16.9.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/5b774a794940bfbbca5c3fb9ddfab1b68bd1c868442bf1e0587fbeafc2c3ba01/json HTTP/1.1" 200 None Stderr:time="2025-04-02T03:58:19Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestprometheusprotocols-gw3_default Creating Stderr: Network roottestprometheusprotocols-gw3_default Created Stderr: Container roottestprometheusprotocols-gw3-prometheus_reader-1 Creating Stderr: Container roottestprometheusprotocols-gw3-prometheus_writer-1 Creating Stderr: Container roottestprometheusprotocols-gw3-prometheus_writer-1 Created Stderr: Container roottestprometheusprotocols-gw3-prometheus_reader-1 Created Stderr: Container roottestprometheusprotocols-gw3-prometheus_reader-1 Starting Stderr: Container roottestprometheusprotocols-gw3-prometheus_writer-1 Starting Stderr: Container roottestprometheusprotocols-gw3-prometheus_reader-1 Started Stderr: Container roottestprometheusprotocols-gw3-prometheus_writer-1 Started Stderr:time="2025-04-02T03:58:19Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:58:19Z" level=debug msg="otel error" error="" Trying to connect to Prometheus... get_instance_ip instance_name=prometheus_reader http://localhost:None "GET /v1.46/containers/roottestprometheusprotocols-gw3-prometheus_reader-1/json HTTP/1.1" 200 None get_instance_ip instance_name=prometheus_writer http://localhost:None "GET /v1.46/containers/roottestprometheusprotocols-gw3-prometheus_writer-1/json HTTP/1.1" 200 None Starting new HTTP connection (1): 172.16.10.2:9091 HTTPConnectionPool(host='172.16.10.2', port=9091): Max retries exceeded with url: /api/v1/query?query=time() (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) Attempt 1 failed, retrying in 2 seconds run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/WithChain.crt', 'https://localhost:8443/'] Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/WithChain.crt https://localhost:8443/] Executing query SELECT getSetting('load_balancing') on node Executing query CREATE ROLE R1, R2 on instance Stdout:Ok. run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/first.crt', 'https://localhost:8443/'] Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/first.crt https://localhost:8443/] http://localhost:None "GET /v1.46/containers/5b774a794940bfbbca5c3fb9ddfab1b68bd1c868442bf1e0587fbeafc2c3ba01/json HTTP/1.1" 200 None Executing query SELECT getSetting('alter_sync') on node Executing query GRANT R1, R2 TO A on instance Connecting to 172.16.8.4(172.16.8.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Exitcode:60 run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['bash', '-c', "openssl s_client -showcerts -servername localhost -connect localhost:8443 /dev/null | grep 'BEGIN CERTIFICATE' | wc -l"] Command:[docker exec roottestreloadcertificate-gw6-node-1 bash -c openssl s_client -showcerts -servername localhost -connect localhost:8443 /dev/null | grep 'BEGIN CERTIFICATE' | wc -l] Executing query SHOW CURRENT ROLES on instance via HTTP interface [gw5] PASSED test_reloading_settings_from_users_xml/test.py::test_force_reload Starting new HTTP connection (1): 172.16.1.2:8123 test_reloading_settings_from_users_xml/test.py::test_reload_on_timeout run container_id:roottestreloadingsettingsfromusersxml-gw5-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjEwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+Zmlyc3Rfb3JfcmFuZG9tPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4yPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml'] Command:[docker exec roottestreloadingsettingsfromusersxml-gw5-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjEwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+Zmlyc3Rfb3JfcmFuZG9tPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4yPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml] http://172.16.1.2:8123 "GET /?session_id=session+%231&query=SHOW+CURRENT+ROLES HTTP/1.1" 200 None Executing query SET DEFAULT ROLE R2 TO A on instance Stdout:2 [gw6] PASSED test_reload_certificate/test.py::test_chain_reload test_reload_certificate/test.py::test_first_than_second_cert run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cat > /etc/clickhouse-server/config.d/cert.xml << EOF\n\n 8443\n \n \n /etc/clickhouse-server/config.d/first.crt\n /etc/clickhouse-server/config.d/first.key\n true\n true\n sslv2,sslv3\n true\n \n \n \n\nEOF'] Command:[docker exec roottestreloadcertificate-gw6-node-1 bash -c cat > /etc/clickhouse-server/config.d/cert.xml << EOF 8443 /etc/clickhouse-server/config.d/first.crt /etc/clickhouse-server/config.d/first.key true true sslv2,sslv3 true EOF] http://localhost:None "GET /v1.46/containers/5b774a794940bfbbca5c3fb9ddfab1b68bd1c868442bf1e0587fbeafc2c3ba01/json HTTP/1.1" 200 None ClickHouse node1 started Executing query SYSTEM RELOAD CONFIG on node Executing query SYSTEM RELOAD CONFIG on node Executing query SHOW CURRENT ROLES on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%231&query=SHOW+CURRENT+ROLES HTTP/1.1" 200 None Executing query SHOW CURRENT ROLES on instance via HTTP interface Executing query drop database if exists pg on node1 Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%232&query=SHOW+CURRENT+ROLES HTTP/1.1" 200 None Executing query DROP USER IF EXISTS A, B on instance [gw0] PASSED test_role/test.py::test_changing_default_roles_affects_new_sessions_only Executing query SELECT getSetting('max_memory_usage') on node run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/first.crt', 'https://localhost:8443/'] Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/first.crt https://localhost:8443/] Executing query create database pg engine = PostgreSQL(postgres1) on node1 Executing query DROP ROLE IF EXISTS R1, R2, R3, R4 on instance Executing query SELECT getSetting('load_balancing') on node Stdout:Ok. run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/second.crt', 'https://localhost:8443/'] Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/second.crt https://localhost:8443/] Executing query show create table pg.test on node1 test_role/test.py::test_combine_privileges Executing query CREATE USER A on instance Executing query SELECT getSetting('alter_sync') on node Exitcode:60 run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cat > /etc/clickhouse-server/config.d/cert.xml << EOF\n\n 8443\n \n \n /etc/clickhouse-server/config.d/second.crt\n /etc/clickhouse-server/config.d/second.key\n true\n true\n sslv2,sslv3\n true\n \n \n \n\nEOF'] Command:[docker exec roottestreloadcertificate-gw6-node-1 bash -c cat > /etc/clickhouse-server/config.d/cert.xml << EOF 8443 /etc/clickhouse-server/config.d/second.crt /etc/clickhouse-server/config.d/second.key true true sslv2,sslv3 true EOF] Executing query detach table pg.test on node1 Executing query CREATE ROLE R1 on instance Executing query SYSTEM RELOAD CONFIG on node Executing query attach table pg.test on node1 Executing query CREATE ROLE R2 on instance run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/second.crt', 'https://localhost:8443/'] Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/second.crt https://localhost:8443/] Connecting to 172.16.4.3(172.16.4.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Stdout:Ok. run container_id:roottestreloadcertificate-gw6-node-1 detach:False nothrow:False cmd: ['curl', '--silent', '--cacert', '/etc/clickhouse-server/config.d/first.crt', 'https://localhost:8443/'] Command:[docker exec roottestreloadcertificate-gw6-node-1 curl --silent --cacert /etc/clickhouse-server/config.d/first.crt https://localhost:8443/] Executing query show create table pg.test on node1 Executing query SELECT * FROM test_table on instance Exitcode:60 Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/.env --project-name roottestreloadcertificate-gw6 --file /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/node/docker-compose.yml stop --timeout 20] [gw6] PASSED test_reload_certificate/test.py::test_first_than_second_cert [gw2] PASSED test_postgresql_database_engine/test.py::test_datetime test_postgresql_database_engine/test.py::test_get_create_table_query_with_multidim_arrays Executing query GRANT R1 TO A on instance Failed connecting to Zookeeper within the connection retry policy. Executing query CREATE DATABASE postgres_database ENGINE = PostgreSQL('postgres1:5432', 'postgres_database', 'postgres', 'mysecretpassword') on node1 Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw4-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.4.2, port:2181, use_ssl:False Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Connecting to 172.16.9.4(172.16.9.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query GRANT SELECT(x) ON test_table TO R1 on instance Executing query DETACH TABLE postgres_database.array_columns on node1 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw4-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.4.4, port:2181, use_ssl:False Connecting to 172.16.4.4(172.16.4.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query SELECT * FROM test_table on instance Executing query ATTACH TABLE postgres_database.array_columns on node1 Executing query SELECT x FROM test_table on instance Executing query INSERT INTO postgres_database.array_columns VALUES ([[[1, 1], [1, 1]], [[3, 3], [3, 3]], [[4, 4], [5, 5]]], [[[1, NULL], [NULL, 1]], [[NULL, NULL], [NULL, NULL]], [[4, 4], [5, 5]]] ) on node1 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') Trying to create Minio instance by command docker compose --project-name roottests3cluster-gw4 --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d Command:[docker compose --project-name roottests3cluster-gw4 --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d] Executing query GRANT SELECT(y) ON test_table TO R2 on instance Executing query SELECT * FROM postgres_database.array_columns on node1 Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query GRANT R2 TO A on instance Executing query DROP DATABASE postgres_database on node1 Executing query SHOW DATABASES on node1 Executing query SELECT * FROM test_table on instance [gw2] PASSED test_postgresql_database_engine/test.py::test_get_create_table_query_with_multidim_arrays test_postgresql_database_engine/test.py::test_inaccessible_postgresql_database_engine_filterable_on_system_tables Executing query CREATE DATABASE postgres_database ENGINE = PostgreSQL('google.com:5432', 'dummy', 'dummy', 'dummy') on node1 Executing query DROP USER IF EXISTS A, B on instance [gw0] PASSED test_role/test.py::test_combine_privileges Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SHOW DATABASES on node1 Executing query DROP ROLE IF EXISTS R1, R2, R3, R4 on instance test_role/test.py::test_create_role Executing query CREATE USER A on instance Connecting to 172.16.8.4(172.16.8.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SELECT DISTINCT(name) FROM system.tables WHERE engine!='PostgreSQL' AND name='COLUMNS' on node1 Executing query CREATE ROLE R1 on instance Executing query SELECT * FROM test_table on instance run container_id:roottestreloadingsettingsfromusersxml-gw5-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjIwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+bmVhcmVzdF9ob3N0bmFtZTwvbG9hZF9iYWxhbmNpbmc+CiAgICAgICAgICAgIDxyZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+MDwvcmVwbGljYXRpb25fYWx0ZXJfcGFydGl0aW9uc19zeW5jPgogICAgICAgIDwvZGVmYXVsdD4KICAgIDwvcHJvZmlsZXM+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/z.xml'] Command:[docker exec roottestreloadingsettingsfromusersxml-gw5-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjIwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+bmVhcmVzdF9ob3N0bmFtZTwvbG9hZF9iYWxhbmNpbmc+CiAgICAgICAgICAgIDxyZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+MDwvcmVwbGljYXRpb25fYWx0ZXJfcGFydGl0aW9uc19zeW5jPgogICAgICAgIDwvZGVmYXVsdD4KICAgIDwvcHJvZmlsZXM+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/z.xml] Executing query GRANT SELECT ON test_table TO R1 on instance Executing query SELECT getSetting('max_memory_usage') on node Executing query SELECT * FROM test_table on instance Stderr:time="2025-04-02T03:58:20Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Volume "roottests3cluster-gw4_data1-1" Creating Stderr: Volume "roottests3cluster-gw4_data1-1" Created Stderr:time="2025-04-02T03:58:20Z" level=warning msg="Found orphan containers ([roottests3cluster-gw4-zoo3-1 roottests3cluster-gw4-zoo1-1 roottests3cluster-gw4-zoo2-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up." Stderr: Container roottests3cluster-gw4-proxy2-1 Creating Stderr: Container roottests3cluster-gw4-proxy1-1 Creating Stderr: Container roottests3cluster-gw4-proxy2-1 Created Stderr: Container roottests3cluster-gw4-proxy1-1 Created Stderr: Container roottests3cluster-gw4-minio1-1 Creating Stderr: Container roottests3cluster-gw4-resolver-1 Creating Stderr: Container roottests3cluster-gw4-resolver-1 Created Stderr: Container roottests3cluster-gw4-minio1-1 Created Stderr: Container roottests3cluster-gw4-proxy1-1 Starting Stderr: Container roottests3cluster-gw4-proxy2-1 Starting Stderr: Container roottests3cluster-gw4-proxy2-1 Started Stderr: Container roottests3cluster-gw4-proxy1-1 Started Stderr: Container roottests3cluster-gw4-minio1-1 Starting Stderr: Container roottests3cluster-gw4-resolver-1 Starting Stderr: Container roottests3cluster-gw4-minio1-1 Started Stderr: Container roottests3cluster-gw4-resolver-1 Started Stderr:time="2025-04-02T03:58:21Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:58:21Z" level=debug msg="otel error" error="" Trying to connect to Minio... get_instance_ip instance_name=minio1 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw4-minio1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=proxy1 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw4-proxy1-1/json HTTP/1.1" 200 None Starting new HTTP connection (1): 172.16.4.7:9001 Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (2): 172.16.4.7:9001 Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (3): 172.16.4.7:9001 Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (4): 172.16.4.7:9001 Can't connect to Minio: HTTPConnectionPool(host='172.16.4.7', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) Executing query GRANT R1 TO A on instance Executing query SELECT * FROM test_table on instance Executing query REVOKE R1 FROM A on instance Executing query SELECT * FROM test_table on instance Executing query DROP USER IF EXISTS A, B on instance [gw0] PASSED test_role/test.py::test_create_role Executing query DROP ROLE IF EXISTS R1, R2, R3, R4 on instance Starting new HTTP connection (1): 172.16.10.2:9091 http://172.16.10.2:9091 "GET /api/v1/query?query=time() HTTP/1.1" 200 104 http://172.16.10.2:9091/api/v1/query?query=time() is available after 2.008305072784424 seconds Starting new HTTP connection (1): 172.16.10.3:9090 http://172.16.10.3:9090 "GET /api/v1/query?query=time() HTTP/1.1" 200 104 http://172.16.10.3:9090/api/v1/query?query=time() is available after 0.003799915313720703 seconds ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/.env --project-name roottestprometheusprotocols-gw3 --file /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_prometheus.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/.env --project-name roottestprometheusprotocols-gw3 --file /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_prometheus.yml up -d --no-recreate] test_role/test.py::test_function_current_roles Executing query CREATE USER A on instance Executing query SELECT getSetting('max_memory_usage') on node Executing query CREATE ROLE R1, R2, R3, R4 on instance Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query GRANT R4 TO R2 on instance Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.7.2, port:2181, use_ssl:False Connecting to 172.16.7.2(172.16.7.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Executing query GRANT R1,R2,R3 TO A on instance Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Connecting to 172.16.9.4(172.16.9.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query SELECT defaultRoles(), currentRoles(), enabledRoles() on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%233&query=SELECT+defaultRoles%28%29%2C+currentRoles%28%29%2C+enabledRoles%28%29 HTTP/1.1" 200 None Executing query SET ROLE R1 on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%233&query=SET+ROLE+R1 HTTP/1.1" 200 None Executing query SELECT defaultRoles(), currentRoles(), enabledRoles() on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%233&query=SELECT+defaultRoles%28%29%2C+currentRoles%28%29%2C+enabledRoles%28%29 HTTP/1.1" 200 None Executing query SET ROLE R2 on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%233&query=SET+ROLE+R2 HTTP/1.1" 200 None Executing query SELECT defaultRoles(), currentRoles(), enabledRoles() on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%233&query=SELECT+defaultRoles%28%29%2C+currentRoles%28%29%2C+enabledRoles%28%29 HTTP/1.1" 200 None Executing query SET ROLE NONE on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%233&query=SET+ROLE+NONE HTTP/1.1" 200 None Executing query SELECT defaultRoles(), currentRoles(), enabledRoles() on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%233&query=SELECT+defaultRoles%28%29%2C+currentRoles%28%29%2C+enabledRoles%28%29 HTTP/1.1" 200 None Executing query SET ROLE DEFAULT on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%233&query=SET+ROLE+DEFAULT HTTP/1.1" 200 None Executing query SELECT defaultRoles(), currentRoles(), enabledRoles() on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%233&query=SELECT+defaultRoles%28%29%2C+currentRoles%28%29%2C+enabledRoles%28%29 HTTP/1.1" 200 None Executing query SET DEFAULT ROLE R2 TO A on instance Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.7.4, port:2181, use_ssl:False Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Sending request(xid=2): Close() get_instance_ip instance_name=zoo2 Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost http://localhost:None "GET /v1.46/containers/roottestreloadclustersconfig-gw7-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.9.3, port:2181, use_ssl:False Connecting to 172.16.9.3(172.16.9.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Stderr: Container roottestprometheusprotocols-gw3-prometheus_writer-1 Running Stderr: Container roottestprometheusprotocols-gw3-prometheus_reader-1 Running Stderr: Container roottestprometheusprotocols-gw3-node-1 Creating Stderr: Container roottestprometheusprotocols-gw3-node-1 Created Stderr: Container roottestprometheusprotocols-gw3-node-1 Starting Stderr: Container roottestprometheusprotocols-gw3-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestprometheusprotocols-gw3-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestprometheusprotocols-gw3-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.10.4... http://localhost:None "GET /v1.46/containers/roottestprometheusprotocols-gw3-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/5501c8bfc41840953a8cd7a63bb8ab891c2f7aaaea7bd6996d2548dea7bf0211/json HTTP/1.1" 200 None Executing query SELECT defaultRoles(), currentRoles(), enabledRoles() on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://172.16.1.2:8123 "GET /?session_id=session+%233&query=SELECT+defaultRoles%28%29%2C+currentRoles%28%29%2C+enabledRoles%28%29 HTTP/1.1" 200 None Executing query REVOKE R3 FROM A on instance http://localhost:None "GET /v1.46/containers/roottestreloadclustersconfig-gw7-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.9.2, port:2181, use_ssl:False Connecting to 172.16.9.2(172.16.9.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Zookeeper connection established, state: CONNECTED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/.env --project-name roottestreplicationcredentials-gw8 --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/.env --project-name roottestreplicationcredentials-gw8 --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/docker-compose.yml up -d --no-recreate] Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query SELECT defaultRoles(), currentRoles(), enabledRoles() on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%233&query=SELECT+defaultRoles%28%29%2C+currentRoles%28%29%2C+enabledRoles%28%29 HTTP/1.1" 200 None Executing query REVOKE R2 FROM A on instance http://localhost:None "GET /v1.46/containers/5501c8bfc41840953a8cd7a63bb8ab891c2f7aaaea7bd6996d2548dea7bf0211/json HTTP/1.1" 200 None Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/.env --project-name roottestreloadclustersconfig-gw7 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/.env --project-name roottestreloadclustersconfig-gw7 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate] Executing query SELECT defaultRoles(), currentRoles(), enabledRoles() on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%233&query=SELECT+defaultRoles%28%29%2C+currentRoles%28%29%2C+enabledRoles%28%29 HTTP/1.1" 200 None Executing query SET DEFAULT ROLE ALL TO A on instance Starting new HTTP connection (5): 172.16.4.7:9001 http://172.16.4.7:9001 "GET / HTTP/1.1" 200 0 Connected to Minio. http://172.16.4.7:9001 "GET /root?location= HTTP/1.1" 404 0 http://172.16.4.7:9001 "PUT /root HTTP/1.1" 200 0 S3 bucket 'root' created http://172.16.4.7:9001 "GET /root2?location= HTTP/1.1" 404 0 http://172.16.4.7:9001 "PUT /root2 HTTP/1.1" 200 0 S3 bucket 'root2' created ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/.env --project-name roottests3cluster-gw4 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_1_0/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/.env --project-name roottests3cluster-gw4 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_1_0/docker-compose.yml up -d --no-recreate] http://localhost:None "GET /v1.46/containers/5501c8bfc41840953a8cd7a63bb8ab891c2f7aaaea7bd6996d2548dea7bf0211/json HTTP/1.1" 200 None Executing query SELECT defaultRoles(), currentRoles(), enabledRoles() on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%233&query=SELECT+defaultRoles%28%29%2C+currentRoles%28%29%2C+enabledRoles%28%29 HTTP/1.1" 200 None Executing query DROP USER IF EXISTS A, B on instance [gw0] PASSED test_role/test.py::test_function_current_roles Executing query SELECT getSetting('max_memory_usage') on node Executing query DROP ROLE IF EXISTS R1, R2, R3, R4 on instance Connecting to 172.16.8.4(172.16.8.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost http://localhost:None "GET /v1.46/containers/5501c8bfc41840953a8cd7a63bb8ab891c2f7aaaea7bd6996d2548dea7bf0211/json HTTP/1.1" 200 None test_role/test.py::test_grant_role_to_role Executing query CREATE USER A on instance Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.8.2, port:2181, use_ssl:False http://localhost:None "GET /v1.46/containers/5501c8bfc41840953a8cd7a63bb8ab891c2f7aaaea7bd6996d2548dea7bf0211/json HTTP/1.1" 200 None ClickHouse node started Connecting to 172.16.8.2(172.16.8.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Executing query CREATE TABLE prometheus (id UInt64) ENGINE=TimeSeries on node Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query CREATE ROLE R1 on instance Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.8.3, port:2181, use_ssl:False Connecting to 172.16.8.3(172.16.8.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') Trying to create Minio instance by command docker compose --project-name roottestremoteblobsnamingbackwardcompatibility-gw1 --env-file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d Command:[docker compose --project-name roottestremoteblobsnamingbackwardcompatibility-gw1 --env-file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d] Executing query CREATE ROLE R2 on instance Starting new HTTP connection (1): 172.16.10.3:9090 http://172.16.10.3:9090 "GET /api/v1/query?query=up&time=1743566303.0500412 HTTP/1.1" 200 87 Executing query SELECT getSetting('max_memory_usage') on node Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Running Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Running Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Running Stderr: Container roottestreloadclustersconfig-gw7-node-1 Creating Stderr: Container roottestreloadclustersconfig-gw7-node-1 Created Stderr: Container roottestreloadclustersconfig-gw7-node-1 Starting Stderr: Container roottestreloadclustersconfig-gw7-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestreloadclustersconfig-gw7-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestreloadclustersconfig-gw7-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.9.5... http://localhost:None "GET /v1.46/containers/roottestreloadclustersconfig-gw7-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Stderr: Container roottestreplicationcredentials-gw8-zoo1-1 Running Stderr: Container roottestreplicationcredentials-gw8-zoo2-1 Running Stderr: Container roottestreplicationcredentials-gw8-zoo3-1 Running Stderr: Container roottestreplicationcredentials-gw8-node5-1 Creating Stderr: Container roottestreplicationcredentials-gw8-node8-1 Creating Stderr: Container roottestreplicationcredentials-gw8-node3-1 Creating Stderr: Container roottestreplicationcredentials-gw8-node7-1 Creating Stderr: Container roottestreplicationcredentials-gw8-node2-1 Creating Stderr: Container roottestreplicationcredentials-gw8-node4-1 Creating Stderr: Container roottestreplicationcredentials-gw8-node1-1 Creating Stderr: Container roottestreplicationcredentials-gw8-node6-1 Creating Stderr: Container roottestreplicationcredentials-gw8-node7-1 Created Stderr: Container roottestreplicationcredentials-gw8-node3-1 Created Stderr: Container roottestreplicationcredentials-gw8-node1-1 Created Stderr: Container roottestreplicationcredentials-gw8-node4-1 Created Stderr: Container roottestreplicationcredentials-gw8-node2-1 Created Stderr: Container roottestreplicationcredentials-gw8-node5-1 Created Stderr: Container roottestreplicationcredentials-gw8-node8-1 Created Stderr: Container roottestreplicationcredentials-gw8-node6-1 Created Stderr: Container roottestreplicationcredentials-gw8-node8-1 Starting Stderr: Container roottestreplicationcredentials-gw8-node6-1 Starting Stderr: Container roottestreplicationcredentials-gw8-node4-1 Starting Stderr: Container roottestreplicationcredentials-gw8-node5-1 Starting Stderr: Container roottestreplicationcredentials-gw8-node3-1 Starting Stderr: Container roottestreplicationcredentials-gw8-node7-1 Starting Stderr: Container roottestreplicationcredentials-gw8-node1-1 Starting Stderr: Container roottestreplicationcredentials-gw8-node2-1 Starting Stderr: Container roottestreplicationcredentials-gw8-node2-1 Started Stderr: Container roottestreplicationcredentials-gw8-node3-1 Started Stderr: Container roottestreplicationcredentials-gw8-node8-1 Started Stderr: Container roottestreplicationcredentials-gw8-node4-1 Started Stderr: Container roottestreplicationcredentials-gw8-node1-1 Started Stderr: Container roottestreplicationcredentials-gw8-node6-1 Started Stderr: Container roottestreplicationcredentials-gw8-node5-1 Started Stderr: Container roottestreplicationcredentials-gw8-node7-1 Started ClickHouse instance created get_instance_ip instance_name=node1 Stderr: Container roottests3cluster-gw4-proxy1-1 Running Stderr: Container roottests3cluster-gw4-zoo1-1 Running Stderr: Container roottests3cluster-gw4-zoo3-1 Running Stderr: Container roottests3cluster-gw4-proxy2-1 Running Stderr: Container roottests3cluster-gw4-minio1-1 Running Stderr: Container roottests3cluster-gw4-resolver-1 Running Stderr: Container roottests3cluster-gw4-zoo2-1 Running Stderr: Container roottests3cluster-gw4-s0_0_0-1 Creating Stderr: Container roottests3cluster-gw4-s0_0_1-1 Creating Stderr: Container roottests3cluster-gw4-s0_1_0-1 Creating Stderr: Container roottests3cluster-gw4-s0_0_1-1 Created Stderr: Container roottests3cluster-gw4-s0_0_0-1 Created Stderr: Container roottests3cluster-gw4-s0_1_0-1 Created Stderr: Container roottests3cluster-gw4-s0_0_1-1 Starting Stderr: Container roottests3cluster-gw4-s0_0_0-1 Starting Stderr: Container roottests3cluster-gw4-s0_1_0-1 Starting Stderr: Container roottests3cluster-gw4-s0_0_0-1 Started Stderr: Container roottests3cluster-gw4-s0_0_1-1 Started Stderr: Container roottests3cluster-gw4-s0_1_0-1 Started ClickHouse instance created get_instance_ip instance_name=s0_0_0 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw4-s0_0_0-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=s0_0_0 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw4-s0_0_0-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in s0_0_0, ip: 172.16.4.10... get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw4-s0_0_0-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b803159ff9ea6c11b73b50a61c8da2c00ab2514e2fd09cf667c6c2ce1504fbcc/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.7.10... http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9ce4fa2057ea1141ba85fba44fe28ecdbb3e2ccf998438cd724eee59b90cd14f/json HTTP/1.1" 200 None Executing query SELECT getSetting('load_balancing') on node Executing query SELECT * FROM test_table on instance Stderr: Container roottestreloadcertificate-gw6-node-1 Stopping Stderr: Container roottestreloadcertificate-gw6-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/.env --project-name roottestreloadcertificate-gw6 --file /ClickHouse/tests/integration/test_reload_certificate/_instances-0-gw6/node/docker-compose.yml down --volumes] http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SELECT getSetting('alter_sync') on node http://localhost:None "GET /v1.46/containers/b803159ff9ea6c11b73b50a61c8da2c00ab2514e2fd09cf667c6c2ce1504fbcc/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9ce4fa2057ea1141ba85fba44fe28ecdbb3e2ccf998438cd724eee59b90cd14f/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Starting new HTTP connection (1): 172.16.10.2:9091 Executing query GRANT R1 TO A on instance [gw5] PASSED test_reloading_settings_from_users_xml/test.py::test_reload_on_timeout test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_enum run container_id:roottestreloadingsettingsfromusersxml-gw5-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjEwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+Zmlyc3Rfb3JfcmFuZG9tPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4yPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml'] Command:[docker exec roottestreloadingsettingsfromusersxml-gw5-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjEwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+Zmlyc3Rfb3JfcmFuZG9tPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4yPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml] http://localhost:None "GET /v1.46/containers/b803159ff9ea6c11b73b50a61c8da2c00ab2514e2fd09cf667c6c2ce1504fbcc/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9ce4fa2057ea1141ba85fba44fe28ecdbb3e2ccf998438cd724eee59b90cd14f/json HTTP/1.1" 200 None http://172.16.10.2:9091 "GET /api/v1/query?query=up&time=1743566303.0500412 HTTP/1.1" 200 87 Executing query SYSTEM RELOAD CONFIG on node Executing query DROP TABLE IF EXISTS prometheus SYNC on node [gw3] PASSED test_prometheus_protocols/test.py::test_64bit_id http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b803159ff9ea6c11b73b50a61c8da2c00ab2514e2fd09cf667c6c2ce1504fbcc/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9ce4fa2057ea1141ba85fba44fe28ecdbb3e2ccf998438cd724eee59b90cd14f/json HTTP/1.1" 200 None Executing query SELECT * FROM test_table on instance Executing query DROP TABLE IF EXISTS original SYNC on node run container_id:roottestreloadingsettingsfromusersxml-gw5-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPmE8L21heF9tZW1vcnlfdXNhZ2U+CiAgICAgICAgICAgIDxsb2FkX2JhbGFuY2luZz5uZWFyZXN0X2hvc3RuYW1lPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4wPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml'] Command:[docker exec roottestreloadingsettingsfromusersxml-gw5-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPmE8L21heF9tZW1vcnlfdXNhZ2U+CiAgICAgICAgICAgIDxsb2FkX2JhbGFuY2luZz5uZWFyZXN0X2hvc3RuYW1lPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4wPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml] http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query GRANT R2 TO R1 on instance http://localhost:None "GET /v1.46/containers/b803159ff9ea6c11b73b50a61c8da2c00ab2514e2fd09cf667c6c2ce1504fbcc/json HTTP/1.1" 200 None ClickHouse s0_0_0 started get_instance_ip instance_name=s0_0_1 Executing query SYSTEM RELOAD CONFIG on node http://localhost:None "GET /v1.46/containers/roottests3cluster-gw4-s0_0_1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=s0_0_1 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw4-s0_0_1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in s0_0_1, ip: 172.16.4.11... http://localhost:None "GET /v1.46/containers/roottests3cluster-gw4-s0_0_1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/9ce4fa2057ea1141ba85fba44fe28ecdbb3e2ccf998438cd724eee59b90cd14f/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/f9cf2e112b8a032d36a93afce47339a6be42f01adb649555b4a0e33e0aaafa45/json HTTP/1.1" 200 None ClickHouse s0_0_1 started get_instance_ip instance_name=s0_1_0 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.7.5... http://localhost:None "GET /v1.46/containers/roottests3cluster-gw4-s0_1_0-1/json HTTP/1.1" 200 None get_instance_ip instance_name=s0_1_0 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottests3cluster-gw4-s0_1_0-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/29e73551c6ee401c84c36adb53adef85e8c8c805a7c863ca3d6f942b248b3bf7/json HTTP/1.1" 200 None Waiting for ClickHouse start in s0_1_0, ip: 172.16.4.9... ClickHouse node2 started get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw4-s0_1_0-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/f3f42846136d6de36c413e69b6dcc25112fc57b9a5435a272f1a606683011468/json HTTP/1.1" 200 None ClickHouse s0_1_0 started get_instance_ip instance_name=node3 Cluster started http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node3, ip: 172.16.7.7... http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node3-1/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS mydata SYNC on node http://localhost:None "GET /v1.46/containers/839c507122d35759dcd8336a105eaa941794c9de07ef4fbb1791170af05c4bd9/json HTTP/1.1" 200 None ClickHouse node3 started get_instance_ip instance_name=node4 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node4-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node4 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node4-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node4, ip: 172.16.7.11... http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node4-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b734901e175dd35d2051e8c61b926a57e826e5d1fb6934866bc18c32a5413691/json HTTP/1.1" 200 None ClickHouse node4 started get_instance_ip instance_name=node5 Executing query SELECT * FROM test_table on instance http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node5-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node5 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node5-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node5, ip: 172.16.7.9... http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node5-1/json HTTP/1.1" 200 None http://172.16.4.7:9001 "PUT /root/data/clickhouse/part1.csv HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/containers/ea69f459f9cc7743c16e1db0d8e9d9db06142047364f4d57f405e45531603367/json HTTP/1.1" 200 None ClickHouse node5 started get_instance_ip instance_name=node6 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node6-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node6 http://172.16.4.7:9001 "PUT /root/data/clickhouse/part123.csv HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node6-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node6, ip: 172.16.7.8... http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node6-1/json HTTP/1.1" 200 None http://172.16.4.7:9001 "PUT /root/data/database/part2.csv HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/containers/325b4b25507843868f0873dac2b6574de1c272c25c52428f8aa8a84198c7bb50/json HTTP/1.1" 200 None ClickHouse node6 started get_instance_ip instance_name=node7 http://172.16.4.7:9001 "PUT /root/data/database/partition675.csv HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node7-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node7 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node7-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node7, ip: 172.16.7.6... http://172.16.4.7:9001 "PUT /root/data/generated/file_0.csv HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node7-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/921d2bc677a7aa9ab267d408e53c45b93fc163f925ea1b5d5897782aad7cfb2c/json HTTP/1.1" 200 None ClickHouse node7 started get_instance_ip instance_name=node8 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None http://172.16.4.7:9001 "PUT /root/data/generated/file_1.csv HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node8-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node8 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node8-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node8, ip: 172.16.7.12... http://172.16.4.7:9001 "PUT /root/data/generated/file_2.csv HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/containers/roottestreplicationcredentials-gw8-node8-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/460fc5dd7dd0b2db8607fef4abec44382cd3d1eb736695326b7a26f22bc5892a/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS mytable SYNC on node Executing query SELECT getSetting('max_memory_usage') on node ClickHouse node8 started http://172.16.4.7:9001 "PUT /root/data/generated/file_3.csv HTTP/1.1" 200 0 Executing query CREATE DATABASE test; CREATE TABLE test_table(date Date, id UInt32, dummy UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test4/replicated', 'node7') PARTITION BY toYYYYMM(date) ORDER BY id; on node7 http://172.16.4.7:9001 "PUT /root/data/generated/file_4.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_5.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_6.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_7.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_8.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_9.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_10.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_11.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_12.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_13.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_14.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_15.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_16.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_17.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_18.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_19.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_20.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_21.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_22.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_23.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_24.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_25.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_26.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_27.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_28.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_29.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_30.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_31.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_32.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_33.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_34.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_35.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_36.csv HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None http://172.16.4.7:9001 "PUT /root/data/generated/file_37.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_38.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_39.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_40.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_41.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_42.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_43.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_44.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_45.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_46.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_47.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_48.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_49.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_50.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_51.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_52.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_53.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_54.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_55.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_56.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_57.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_58.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_59.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_60.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_61.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_62.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_63.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_64.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_65.csv HTTP/1.1" 200 0 Stderr: Container roottestreloadcertificate-gw6-node-1 Stopping Stderr: Container roottestreloadcertificate-gw6-node-1 Stopped Stderr: Container roottestreloadcertificate-gw6-node-1 Removing Stderr: Container roottestreloadcertificate-gw6-node-1 Removed Stderr: Network roottestreloadcertificate-gw6_default Removing Stderr: Network roottestreloadcertificate-gw6_default Removed Cleanup called http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None http://172.16.4.7:9001 "PUT /root/data/generated/file_66.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_67.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_68.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_69.csv HTTP/1.1" 200 0 Docker networks for project roottestreloadcertificate-gw6 are NETWORK ID NAME DRIVER SCOPE Executing query DROP TABLE IF EXISTS mymetrics SYNC on node http://172.16.4.7:9001 "PUT /root/data/generated/file_70.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_71.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_72.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_73.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_74.csv HTTP/1.1" 200 0 Executing query GRANT SELECT ON test_table TO R2 on instance http://172.16.4.7:9001 "PUT /root/data/generated/file_75.csv HTTP/1.1" 200 0 Docker containers for project roottestreloadcertificate-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES http://172.16.4.7:9001 "PUT /root/data/generated/file_76.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_77.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_78.csv HTTP/1.1" 200 0 Stderr:time="2025-04-02T03:58:23Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Volume "roottestremoteblobsnamingbackwardcompatibility-gw1_data1-1" Creating Stderr: Volume "roottestremoteblobsnamingbackwardcompatibility-gw1_data1-1" Created Stderr:time="2025-04-02T03:58:23Z" level=warning msg="Found orphan containers ([roottestremoteblobsnamingbackwardcompatibility-gw1-zoo2-1 roottestremoteblobsnamingbackwardcompatibility-gw1-zoo1-1 roottestremoteblobsnamingbackwardcompatibility-gw1-zoo3-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up." Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy2-1 Creating Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy1-1 Creating Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy2-1 Created Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy1-1 Created Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-minio1-1 Creating Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-resolver-1 Creating Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-resolver-1 Created Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-minio1-1 Created Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy2-1 Starting Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy1-1 Starting Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy1-1 Started Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy2-1 Started Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-minio1-1 Starting Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-resolver-1 Starting Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-minio1-1 Started Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-resolver-1 Started Stderr:time="2025-04-02T03:58:24Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:58:24Z" level=debug msg="otel error" error="" Trying to connect to Minio... get_instance_ip instance_name=minio1 http://172.16.4.7:9001 "PUT /root/data/generated/file_79.csv HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-minio1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=proxy1 http://localhost:None "GET /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-proxy1-1/json HTTP/1.1" 200 None Starting new HTTP connection (1): 172.16.8.8:9001 Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / http://172.16.4.7:9001 "PUT /root/data/generated/file_80.csv HTTP/1.1" 200 0 Starting new HTTP connection (2): 172.16.8.8:9001 Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (3): 172.16.8.8:9001 Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (4): 172.16.8.8:9001 Can't connect to Minio: HTTPConnectionPool(host='172.16.8.8', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) http://172.16.4.7:9001 "PUT /root/data/generated/file_81.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_82.csv HTTP/1.1" 200 0 Docker volumes for project roottestreloadcertificate-gw6 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreloadcertificate-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] http://172.16.4.7:9001 "PUT /root/data/generated/file_83.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_84.csv HTTP/1.1" 200 0 Executing query SELECT getSetting('load_balancing') on node Executing query CREATE DATABASE test; CREATE TABLE test_table(date Date, id UInt32, dummy UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test4/replicated', 'node8') PARTITION BY toYYYYMM(date) ORDER BY id; on node8 http://172.16.4.7:9001 "PUT /root/data/generated/file_85.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_86.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_87.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_88.csv HTTP/1.1" 200 0 Unstopped containers: {} No running containers for project: roottestreloadcertificate-gw6 http://172.16.4.7:9001 "PUT /root/data/generated/file_89.csv HTTP/1.1" 200 0 test_prometheus_protocols/test.py::test_create_as_table Executing query CREATE TABLE original ENGINE=TimeSeries on node http://172.16.4.7:9001 "PUT /root/data/generated/file_90.csv HTTP/1.1" 200 0 Trying to prune unused networks... http://172.16.4.7:9001 "PUT /root/data/generated/file_91.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_92.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_93.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_94.csv HTTP/1.1" 200 0 Executing query SELECT * FROM test_table on instance http://172.16.4.7:9001 "PUT /root/data/generated/file_95.csv HTTP/1.1" 200 0 Trying to prune unused images... http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Command:[docker image prune -f] http://172.16.4.7:9001 "PUT /root/data/generated/file_96.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_97.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_98.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "PUT /root/data/generated/file_99.csv HTTP/1.1" 200 0 http://172.16.4.7:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1" 200 0 Starting mock server s3_mock.py run container_id:roottests3cluster-gw4-resolver-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname s3_mock.py) && echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJlcXVlc3QsIHJlc3BvbnNlLCByb3V0ZSwgcnVuCgoKQHJvdXRlKCIvPF9idWNrZXQ+LzxfcGF0aDpwYXRoPiIpCmRlZiBzZXJ2ZXIoX2J1Y2tldCwgX3BhdGgpOgogICAgcmVzdWx0ID0gKAogICAgICAgIHJlcXVlc3QuaGVhZGVyc1siTXlDdXN0b21IZWFkZXIiXQogICAgICAgIGlmICJNeUN1c3RvbUhlYWRlciIgaW4gcmVxdWVzdC5oZWFkZXJzCiAgICAgICAgZWxzZSAidW5rbm93biIKICAgICkKICAgIHJlc3BvbnNlLmNvbnRlbnRfdHlwZSA9ICJ0ZXh0L3BsYWluIgogICAgcmVzcG9uc2Uuc2V0X2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4ocmVzdWx0KSkKICAgIHJldHVybiByZXN1bHQKCgpAcm91dGUoIi8iKQpkZWYgcGluZygpOgogICAgcmVzcG9uc2UuY29udGVudF90eXBlID0gInRleHQvcGxhaW4iCiAgICByZXNwb25zZS5zZXRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIDIpCiAgICByZXR1cm4gIk9LIgoKCnJ1bihob3N0PSIwLjAuMC4wIiwgcG9ydD1pbnQoc3lzLmFyZ3ZbMV0pKQo= | base64 --decode > s3_mock.py'] Command:[docker exec roottests3cluster-gw4-resolver-1 bash -c mkdir -p $(dirname s3_mock.py) && echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJlcXVlc3QsIHJlc3BvbnNlLCByb3V0ZSwgcnVuCgoKQHJvdXRlKCIvPF9idWNrZXQ+LzxfcGF0aDpwYXRoPiIpCmRlZiBzZXJ2ZXIoX2J1Y2tldCwgX3BhdGgpOgogICAgcmVzdWx0ID0gKAogICAgICAgIHJlcXVlc3QuaGVhZGVyc1siTXlDdXN0b21IZWFkZXIiXQogICAgICAgIGlmICJNeUN1c3RvbUhlYWRlciIgaW4gcmVxdWVzdC5oZWFkZXJzCiAgICAgICAgZWxzZSAidW5rbm93biIKICAgICkKICAgIHJlc3BvbnNlLmNvbnRlbnRfdHlwZSA9ICJ0ZXh0L3BsYWluIgogICAgcmVzcG9uc2Uuc2V0X2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4ocmVzdWx0KSkKICAgIHJldHVybiByZXN1bHQKCgpAcm91dGUoIi8iKQpkZWYgcGluZygpOgogICAgcmVzcG9uc2UuY29udGVudF90eXBlID0gInRleHQvcGxhaW4iCiAgICByZXNwb25zZS5zZXRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIDIpCiAgICByZXR1cm4gIk9LIgoKCnJ1bihob3N0PSIwLjAuMC4wIiwgcG9ydD1pbnQoc3lzLmFyZ3ZbMV0pKQo= | base64 --decode > s3_mock.py] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query SELECT getSetting('alter_sync') on node Stdout:8 Command:[docker volume prune -f] Executing query CREATE TABLE prometheus AS original on node Stdout:Total reclaimed space: 0B Volumes pruned: 8 Executing query DROP USER IF EXISTS A, B on instance [gw0] PASSED test_role/test.py::test_grant_role_to_role test_recompression_ttl/test.py::test_recompression_multiple_ttls Running tests in /ClickHouse/tests/integration/test_recompression_ttl/test.py Cluster start called. is_up=False Executing query insert into test_table values ('2017-06-21', 111, 0) on node7 Docker networks for project roottestrecompressionttl-gw6 are NETWORK ID NAME DRIVER SCOPE run container_id:roottests3cluster-gw4-resolver-1 detach:True nothrow:False cmd: ['bash', '-c', 'python3 s3_mock.py 8080 >/var/log/resolver/s3_mock.log 2>/var/log/resolver/s3_mock.err.log'] Command:[docker exec roottests3cluster-gw4-resolver-1 bash -c python3 s3_mock.py 8080 >/var/log/resolver/s3_mock.log 2>/var/log/resolver/s3_mock.err.log] run container_id:roottests3cluster-gw4-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] Command:[docker exec roottests3cluster-gw4-resolver-1 curl -s http://localhost:8080/] [gw5] PASSED test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_enum test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_int run container_id:roottestreloadingsettingsfromusersxml-gw5-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjEwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+Zmlyc3Rfb3JfcmFuZG9tPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4yPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml'] Command:[docker exec roottestreloadingsettingsfromusersxml-gw5-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjEwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+Zmlyc3Rfb3JfcmFuZG9tPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4yPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml] Docker containers for project roottestrecompressionttl-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Starting new HTTP connection (1): 172.16.10.3:9090 http://172.16.10.3:9090 "GET /api/v1/query?query=up&time=1743566304.7613904 HTTP/1.1" 200 87 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Docker volumes for project roottestrecompressionttl-gw6 are DRIVER VOLUME NAME Cleanup called Executing query DROP ROLE IF EXISTS R1, R2, R3, R4 on instance Executing query SYSTEM RELOAD CONFIG on node Docker networks for project roottestrecompressionttl-gw6 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrecompressionttl-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES test_role/test.py::test_introspection Executing query CREATE USER A on instance Exitcode:7 Docker volumes for project roottestrecompressionttl-gw6 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrecompressionttl-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestrecompressionttl-gw6 Trying to prune unused networks... run container_id:roottestreloadingsettingsfromusersxml-gw5-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPmE8L21heF9tZW1vcnlfdXNhZ2U+CiAgICAgICAgICAgIDxsb2FkX2JhbGFuY2luZz5uZWFyZXN0X2hvc3RuYW1lPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4wPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml'] Command:[docker exec roottestreloadingsettingsfromusersxml-gw5-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPmE8L21heF9tZW1vcnlfdXNhZ2U+CiAgICAgICAgICAgIDxsb2FkX2JhbGFuY2luZz5uZWFyZXN0X2hvc3RuYW1lPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4wPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml] Trying to prune unused images... Command:[docker image prune -f] http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query CREATE USER B on instance Stdout:8 Command:[docker volume prune -f] Executing query SYSTEM RELOAD CONFIG on node Stdout:Total reclaimed space: 0B Volumes pruned: 8 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_recompression_ttl/configs/background_pool_config.xml'] to /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node1/database Setup logs dir /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_recompression_ttl/configs/background_pool_config.xml'] to /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node2/database Setup logs dir /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/.env --project-name roottestrecompressionttl-gw6 --file /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node2/docker-compose.yml pull] Executing query CREATE ROLE R1 on instance Executing query SELECT getSetting('max_memory_usage') on node Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query CREATE ROLE R2 on instance Executing query SELECT getSetting('load_balancing') on node Executing query GRANT R1 TO A on instance http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestrestorereplica-gw9-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.5.4, port:2181, use_ssl:False Connecting to 172.16.5.4(172.16.5.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query SELECT getSetting('alter_sync') on node Executing query GRANT R2 TO B WITH ADMIN OPTION on instance [gw5] PASSED test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_int test_reloading_settings_from_users_xml/test.py::test_unknown_setting_force_reload run container_id:roottestreloadingsettingsfromusersxml-gw5-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjEwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+Zmlyc3Rfb3JfcmFuZG9tPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4yPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml'] Command:[docker exec roottestreloadingsettingsfromusersxml-gw5-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjEwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+Zmlyc3Rfb3JfcmFuZG9tPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4yPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml] http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestrestorereplica-gw9-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.5.2, port:2181, use_ssl:False Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query GRANT SELECT ON test.table TO A, R2 on instance Executing query SYSTEM RELOAD CONFIG on node Executing query GRANT CREATE ON *.* TO B WITH GRANT OPTION on instance run container_id:roottestreloadingsettingsfromusersxml-gw5-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDx4eXo+ODwveHl6PgogICAgICAgIDwvZGVmYXVsdD4KICAgIDwvcHJvZmlsZXM+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/z.xml'] Command:[docker exec roottestreloadingsettingsfromusersxml-gw5-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDx4eXo+ODwveHl6PgogICAgICAgIDwvZGVmYXVsdD4KICAgIDwvcHJvZmlsZXM+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/z.xml] Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/.env --project-name roottestrestorereplica-gw9 --file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica2/docker-compose.yml --file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica3/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/.env --project-name roottestrestorereplica-gw9 --file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica2/docker-compose.yml --file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica3/docker-compose.yml up -d --no-recreate] http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SYSTEM RELOAD CONFIG on node Executing query REVOKE SELECT(x) ON test.table FROM R2 on instance http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SELECT getSetting('max_memory_usage') on node Executing query SHOW ROLES on instance Executing query SELECT getSetting('load_balancing') on node Executing query SHOW CREATE ROLE R1 on instance http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SELECT getSetting('alter_sync') on node Executing query SHOW CREATE ROLE R2 on instance [gw5] PASSED test_reloading_settings_from_users_xml/test.py::test_unknown_setting_force_reload test_reloading_settings_from_users_xml/test.py::test_unknown_setting_reload_on_timeout run container_id:roottestreloadingsettingsfromusersxml-gw5-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjEwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+Zmlyc3Rfb3JfcmFuZG9tPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4yPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml'] Command:[docker exec roottestreloadingsettingsfromusersxml-gw5-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDxtYXhfbWVtb3J5X3VzYWdlPjEwMDAwMDAwMDAwPC9tYXhfbWVtb3J5X3VzYWdlPgogICAgICAgICAgICA8bG9hZF9iYWxhbmNpbmc+Zmlyc3Rfb3JfcmFuZG9tPC9sb2FkX2JhbGFuY2luZz4KICAgICAgICAgICAgPHJlcGxpY2F0aW9uX2FsdGVyX3BhcnRpdGlvbnNfc3luYz4yPC9yZXBsaWNhdGlvbl9hbHRlcl9wYXJ0aXRpb25zX3N5bmM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC9wcm9maWxlcz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/z.xml] http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Starting new HTTP connection (5): 172.16.8.8:9001 http://172.16.8.8:9001 "GET / HTTP/1.1" 200 0 Connected to Minio. http://172.16.8.8:9001 "GET /root?location= HTTP/1.1" 404 0 http://172.16.8.8:9001 "PUT /root HTTP/1.1" 200 0 S3 bucket 'root' created http://172.16.8.8:9001 "GET /root2?location= HTTP/1.1" 404 0 http://172.16.8.8:9001 "PUT /root2 HTTP/1.1" 200 0 S3 bucket 'root2' created ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/.env --project-name roottestremoteblobsnamingbackwardcompatibility-gw1 --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/new_node/docker-compose.yml --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/switching_node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/.env --project-name roottestremoteblobsnamingbackwardcompatibility-gw1 --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/new_node/docker-compose.yml --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/switching_node/docker-compose.yml up -d --no-recreate] Executing query SYSTEM RELOAD CONFIG on node Executing query SHOW CREATE ROLES R1, R2 on instance http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Starting new HTTP connection (1): 172.16.10.2:9091 http://172.16.10.2:9091 "GET /api/v1/query?query=up&time=1743566304.7613904 HTTP/1.1" 200 87 Executing query DROP TABLE IF EXISTS prometheus SYNC on node [gw3] PASSED test_prometheus_protocols/test.py::test_create_as_table Executing query SELECT id FROM test_table order by id on node7 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Stderr: Container roottestrestorereplica-gw9-zoo1-1 Running Stderr: Container roottestrestorereplica-gw9-zoo3-1 Running Stderr: Container roottestrestorereplica-gw9-zoo2-1 Running Stderr: Container roottestrestorereplica-gw9-replica1-1 Creating Stderr: Container roottestrestorereplica-gw9-replica2-1 Creating Stderr: Container roottestrestorereplica-gw9-replica3-1 Creating Stderr: Container roottestrestorereplica-gw9-replica2-1 Created Stderr: Container roottestrestorereplica-gw9-replica3-1 Created Stderr: Container roottestrestorereplica-gw9-replica1-1 Created Stderr: Container roottestrestorereplica-gw9-replica1-1 Starting Stderr: Container roottestrestorereplica-gw9-replica3-1 Starting Stderr: Container roottestrestorereplica-gw9-replica2-1 Starting Stderr: Container roottestrestorereplica-gw9-replica3-1 Started Stderr: Container roottestrestorereplica-gw9-replica1-1 Started Stderr: Container roottestrestorereplica-gw9-replica2-1 Started ClickHouse instance created get_instance_ip instance_name=replica1 http://localhost:None "GET /v1.46/containers/roottestrestorereplica-gw9-replica1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=replica1 http://localhost:None "GET /v1.46/containers/roottestrestorereplica-gw9-replica1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in replica1, ip: 172.16.5.7... http://localhost:None "GET /v1.46/containers/roottestrestorereplica-gw9-replica1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/260117201e1f9cdbf82aeedecba448600c677f98c1e144e9f8806e440fd44f1d/json HTTP/1.1" 200 None Executing query SHOW CREATE ROLES on instance run container_id:roottests3cluster-gw4-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] Command:[docker exec roottests3cluster-gw4-resolver-1 curl -s http://localhost:8080/] Executing query DROP TABLE IF EXISTS original SYNC on node Executing query SELECT id FROM test_table order by id on node8 Executing query SHOW GRANTS FOR A on instance Stdout:OK s3_mock.py answered OK on attempt 2 Mock server s3_mock.py started Executing query SELECT l.name, r.value from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') as l JOIN s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') as r ON l.name = r.name on s0_0_0 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/260117201e1f9cdbf82aeedecba448600c677f98c1e144e9f8806e440fd44f1d/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS mydata SYNC on node Executing query SHOW GRANTS FOR B on instance Executing query insert into test_table values ('2017-06-22', 222, 1) on node8 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/260117201e1f9cdbf82aeedecba448600c677f98c1e144e9f8806e440fd44f1d/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/260117201e1f9cdbf82aeedecba448600c677f98c1e144e9f8806e440fd44f1d/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS mytable SYNC on node Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy2-1 Running Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy1-1 Running Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo2-1 Running Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo1-1 Running Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo3-1 Running Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-minio1-1 Running Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-resolver-1 Running Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 Creating Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-node-1 Creating Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1 Creating Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-node-1 Created Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1 Created Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 Created Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 Starting Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1 Starting Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-node-1 Starting Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-node-1 Started Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1 Started Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.8.11... http://localhost:None "GET /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0ecc4139d2603abda11f6f8f5dfff40ede095ad2e5373bd7eb46f5ba6690effb/json HTTP/1.1" 200 None Executing query SHOW GRANTS FOR R1 on instance http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS mymetrics SYNC on node http://localhost:None "GET /v1.46/containers/260117201e1f9cdbf82aeedecba448600c677f98c1e144e9f8806e440fd44f1d/json HTTP/1.1" 200 None Executing query SHOW GRANTS FOR R2 on instance Executing query CREATE TABLE prometheus (id FixedString(16) DEFAULT murmurHash3_128(metric_name, all_tags)) ENGINE=TimeSeries on node test_prometheus_protocols/test.py::test_custom_id_algorithm http://localhost:None "GET /v1.46/containers/0ecc4139d2603abda11f6f8f5dfff40ede095ad2e5373bd7eb46f5ba6690effb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/260117201e1f9cdbf82aeedecba448600c677f98c1e144e9f8806e440fd44f1d/json HTTP/1.1" 200 None ClickHouse replica1 started get_instance_ip instance_name=replica2 http://localhost:None "GET /v1.46/containers/roottestrestorereplica-gw9-replica2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=replica2 http://localhost:None "GET /v1.46/containers/roottestrestorereplica-gw9-replica2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in replica2, ip: 172.16.5.6... http://localhost:None "GET /v1.46/containers/roottestrestorereplica-gw9-replica2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/694a34aafd4d9728a6348392410c11fb11508309d0c1a0c1d1f0d4fefb17f6b4/json HTTP/1.1" 200 None ClickHouse replica2 started get_instance_ip instance_name=replica3 http://localhost:None "GET /v1.46/containers/roottestrestorereplica-gw9-replica3-1/json HTTP/1.1" 200 None get_instance_ip instance_name=replica3 http://localhost:None "GET /v1.46/containers/roottestrestorereplica-gw9-replica3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in replica3, ip: 172.16.5.5... http://localhost:None "GET /v1.46/containers/roottestrestorereplica-gw9-replica3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/04d1b1b55418d3a1c7c1e41e65c6386039dfa30f5de3c04b7b6cabd7c17220a9/json HTTP/1.1" 200 None ClickHouse replica3 started Executing query DROP TABLE IF EXISTS test SYNC on replica1 Executing query SHOW GRANTS on instance Starting new HTTP connection (1): 172.16.10.3:9090 http://172.16.10.3:9090 "GET /api/v1/query?query=up&time=1743566306.3580425 HTTP/1.1" 200 162 [gw4] PASSED test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression Executing query insert into function s3('http://minio1:9001/root/data/data1', 'minio', 'minio123', JSONEachRow) select 1 as id settings s3_truncate_on_insert=1 on s0_0_0 Executing query DROP TABLE IF EXISTS test SYNC on replica2 http://localhost:None "GET /v1.46/containers/0ecc4139d2603abda11f6f8f5dfff40ede095ad2e5373bd7eb46f5ba6690effb/json HTTP/1.1" 200 None Executing query SHOW GRANTS FOR R1 on instance http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query insert into function s3('http://minio1:9001/root/data/data2', 'minio', 'minio123', JSONEachRow) select * from numbers(0) settings s3_truncate_on_insert=1 on s0_0_0 Executing query DROP TABLE IF EXISTS test SYNC on replica3 Executing query SHOW GRANTS FOR R2 on instance Executing query insert into function s3('http://minio1:9001/root/data/data3', 'minio', 'minio123', JSONEachRow) select 2 as id settings s3_truncate_on_insert=1 on s0_0_0 http://localhost:None "GET /v1.46/containers/0ecc4139d2603abda11f6f8f5dfff40ede095ad2e5373bd7eb46f5ba6690effb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query CREATE TABLE test(n UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/', 'replica1') ORDER BY n PARTITION BY n % 10; on replica1 Executing query SHOW GRANTS on instance Executing query SELECT * FROM s3('http://minio1:9001/root/data/data{1,2,3}', 'minio', 'minio123', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 Executing query CREATE TABLE test(n UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/', 'replica2') ORDER BY n PARTITION BY n % 10; on replica2 Executing query SHOW CURRENT ROLES on instance http://localhost:None "GET /v1.46/containers/0ecc4139d2603abda11f6f8f5dfff40ede095ad2e5373bd7eb46f5ba6690effb/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'minio123', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 Executing query CREATE TABLE test(n UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/', 'replica3') ORDER BY n PARTITION BY n % 10; on replica3 Executing query SHOW CURRENT ROLES on instance http://localhost:None "GET /v1.46/containers/0ecc4139d2603abda11f6f8f5dfff40ede095ad2e5373bd7eb46f5ba6690effb/json HTTP/1.1" 200 None ClickHouse node started get_instance_ip instance_name=new_node http://localhost:None "GET /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=new_node http://localhost:None "GET /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in new_node, ip: 172.16.8.10... http://localhost:None "GET /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c5d3f02275550dc06f0bd3e3d0ae467323324e0f8e40e135f57bf4440ca2f5ed/json HTTP/1.1" 200 None ClickHouse new_node started get_instance_ip instance_name=switching_node http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=switching_node http://localhost:None "GET /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in switching_node, ip: 172.16.8.9... http://localhost:None "GET /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4b7c99d9fcef40f9e4c769de77f0404ad31db021757151bf4e6ca0502ccb8a9d/json HTTP/1.1" 200 None ClickHouse switching_node started Cluster started Executing query CREATE TABLE test_log_table ( id Int64, val String ) ENGINE=Log SETTINGS storage_policy='s3' on switching_node Executing query SHOW ENABLED ROLES on instance Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'minio123', 'auto', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 get_instance_ip instance_name=zoo1 Executing query INSERT INTO test_log_table VALUES (0, 'a') on switching_node Executing query SHOW ENABLED ROLES on instance run container_id:roottestreloadingsettingsfromusersxml-gw5-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDx4eXo+ODwveHl6PgogICAgICAgIDwvZGVmYXVsdD4KICAgIDwvcHJvZmlsZXM+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/z.xml'] Command:[docker exec roottestreloadingsettingsfromusersxml-gw5-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/users.d/z.xml) && echo PGNsaWNraG91c2U+CiAgICA8cHJvZmlsZXMgcmVwbGFjZT0icmVwbGFjZSI+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDx4eXo+ODwveHl6PgogICAgICAgIDwvZGVmYXVsdD4KICAgIDwvcHJvZmlsZXM+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/z.xml] http://localhost:None "GET /v1.46/containers/roottestrestorereplica-gw9-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.5.3, port:2181, use_ssl:False Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Executing query DROP TABLE IF EXISTS test SYNC on replica1 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None run container_id:roottestreloadingsettingsfromusersxml-gw5-node-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "Setting xyz is neither a builtin setting nor started with the prefix \'custom_\' registered for user-defined settings" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestreloadingsettingsfromusersxml-gw5-node-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "Setting xyz is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'minio123', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id on s0_0_0 Executing query SELECT count() FROM test_log_table on switching_node Executing query SHOW ACCESS on instance Executing query DROP TABLE IF EXISTS test SYNC on replica2 Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 03:58:25.372705 [ 11 ] {9e5040f6-ebc8-460c-9107-421ff89ef4c3} executeQuery: Code: 347. DB::Exception: Code: 115. DB::Exception: Setting xyz is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings: while parsing profile 'default' in users configuration file: while loading configuration file '/etc/clickhouse-server/users.xml'. (UNKNOWN_SETTING), Stack trace (when copying this message, always include the lines below): Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 03:58:25.372918 [ 11 ] {} TCPHandler: Code: 347. DB::Exception: Code: 115. DB::Exception: Setting xyz is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings: while parsing profile 'default' in users configuration file: while loading configuration file '/etc/clickhouse-server/users.xml'. (UNKNOWN_SETTING), Stack trace (when copying this message, always include the lines below): Executing query SELECT getSetting('max_memory_usage') on node Executing query SHOW ACCESS on instance http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS test SYNC on replica3 Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'minio123', 'auto', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id on s0_0_0 Executing query SELECT getSetting('load_balancing') on node run container_id:roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cat /etc/clickhouse-server/config.d/switching_node.xml'] Command:[docker exec roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 bash -c cat /etc/clickhouse-server/config.d/switching_node.xml] Executing query SHOW ACCESS on instance Stdout: Stdout: Stdout: Stdout: 0 Stdout: run container_id:roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/0101\n 9009\n \n admin\n 222\n true\n \n \n ' > /etc/clickhouse-server/config.d/credentials1.xml"] Command:[docker exec roottestreplicationcredentials-gw8-node7-1 bash -c echo ' 9009 admin 222 true ' > /etc/clickhouse-server/config.d/credentials1.xml] Executing query SELECT sum(n), count() FROM test on replica2 Executing query SYSTEM RELOAD CONFIG on node7 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SELECT * from system.enabled_roles ORDER BY role_name on instance Executing query insert into test_table values ('2017-06-22', 333, 1) on node7 Executing query SELECT sum(n), count() FROM test on replica3 Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/generated/*', 'minio', 'minio123') order by c1, c2 on s0_0_0 Executing query DROP USER IF EXISTS A, B on instance [gw0] PASSED test_role/test.py::test_introspection http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SYSTEM SYNC REPLICA test_table on node8 Executing query DROP ROLE IF EXISTS R1, R2, R3, R4 on instance Executing query SELECT id FROM test_table order by id on node8 Executing query INSERT INTO test SELECT number + 0 FROM numbers(200) on replica1 test_role/test.py::test_revoke_requires_admin_option Executing query CREATE USER A, B on instance http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/generated/*', 'minio', 'minio123', auto, 'a String, b UInt64') order by a, b on s0_0_0 [gw8] PASSED test_replication_credentials/test.py::test_credentials_and_no_credentials Running tests in /ClickHouse/tests/integration/test_replication_credentials/test.py test_replication_credentials/test.py::test_different_credentials Instance directory already exists. Did you call cluster.start() for second time? Cluster start called. is_up=True Docker networks for project roottestreplicationcredentials-gw8 are NETWORK ID NAME DRIVER SCOPE 21b6fb026e24 roottestreplicationcredentials-gw8_default bridge local Docker containers for project roottestreplicationcredentials-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 325b4b255078 altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 5 seconds ago Up 5 seconds roottestreplicationcredentials-gw8-node6-1 29e73551c6ee altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 5 seconds ago Up 5 seconds roottestreplicationcredentials-gw8-node2-1 9ce4fa2057ea altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 5 seconds ago Up 5 seconds roottestreplicationcredentials-gw8-node1-1 921d2bc677a7 altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 5 seconds ago Up 5 seconds roottestreplicationcredentials-gw8-node7-1 839c507122d3 altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 5 seconds ago Up 5 seconds roottestreplicationcredentials-gw8-node3-1 b734901e175d altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 5 seconds ago Up 5 seconds roottestreplicationcredentials-gw8-node4-1 460fc5dd7dd0 altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 5 seconds ago Up 4 seconds roottestreplicationcredentials-gw8-node8-1 ea69f459f9cc altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 5 seconds ago Up 5 seconds roottestreplicationcredentials-gw8-node5-1 6f1d895347f4 altinityinfra/integration-test:8b2301119731 "clickhouse keeper -…" 9 seconds ago Up 9 seconds roottestreplicationcredentials-gw8-zoo1-1 a1852004e00d altinityinfra/integration-test:8b2301119731 "clickhouse keeper -…" 9 seconds ago Up 9 seconds roottestreplicationcredentials-gw8-zoo3-1 2467e2a28494 altinityinfra/integration-test:8b2301119731 "clickhouse keeper -…" 9 seconds ago Up 9 seconds roottestreplicationcredentials-gw8-zoo2-1 Executing query CREATE ROLE R1, R2 on instance Docker volumes for project roottestreplicationcredentials-gw8 are DRIVER VOLUME NAME Executing query CREATE DATABASE test; CREATE TABLE test_table(date Date, id UInt32, dummy UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test3/replicated', 'node5') PARTITION BY toYYYYMM(date) ORDER BY id; on node5 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query INSERT INTO test SELECT number + 200 FROM numbers(200) on replica1 Executing query GRANT R1 TO B on instance Executing query CREATE DATABASE test; CREATE TABLE test_table(date Date, id UInt32, dummy UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test3/replicated', 'node6') PARTITION BY toYYYYMM(date) ORDER BY id; on node6 Executing query SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) on s0_0_0 [gw4] PASSED test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_with_header Executing query SHOW GRANTS FOR B on instance Executing query insert into test_table values ('2017-06-20', 111, 0) on node5 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query REVOKE R1 FROM B on instance Executing query INSERT INTO test SELECT number + 400 FROM numbers(200) on replica1 Executing query SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'), 'CSV') on s0_0_0 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SHOW GRANTS FOR B on instance Executing query SELECT * from s3Cluster('cluster_simple', 'http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) on s0_0_0 Executing query GRANT R1 TO A on instance Executing query INSERT INTO test SELECT number + 600 FROM numbers(200) on replica1 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query REVOKE R1 FROM B on instance Executing query SELECT * from s3Cluster('cluster_simple', 'http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'), 'CSV') on s0_0_0 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SHOW GRANTS FOR B on instance Executing query INSERT INTO test SELECT number + 800 FROM numbers(200) on replica1 [gw4] PASSED test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection Executing query SELECT * from s3(test_s3) ORDER BY (c1, c2, c3) on s0_0_0 run container_id:roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query GRANT R1 TO A WITH ADMIN OPTION on instance Stdout:9 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SELECT sum(n), count() FROM test on replica1 Executing query REVOKE R1 FROM B on instance Executing query SELECT * from s3Cluster(cluster_simple, test_s3) ORDER BY (c1, c2, c3) on s0_0_0 Executing query SHOW GRANTS FOR B on instance http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Starting new HTTP connection (1): 172.16.10.2:9091 http://172.16.10.2:9091 "GET /api/v1/query?query=up&time=1743566306.3580425 HTTP/1.1" 200 162 Executing query DROP TABLE IF EXISTS prometheus SYNC on node [gw3] PASSED test_prometheus_protocols/test.py::test_custom_id_algorithm Executing query SELECT sum(n), count() FROM test on replica2 Executing query SELECT * from s3Cluster(cluster_simple, test_s3, structure='auto') ORDER BY (c1, c2, c3) on s0_0_0 Executing query GRANT R1 TO B on instance Executing query DROP TABLE IF EXISTS original SYNC on node Executing query SELECT sum(n), count() FROM test on replica3 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS mydata SYNC on node Executing query SHOW GRANTS FOR B on instance Sending request(xid=1): GetChildren(path='/clickhouse/tables/test/replicas/replica2', watcher=None) Received response(xid=1): ['flags', 'is_lost', 'metadata', 'max_processed_insert_time', 'queue', 'parts', 'log_pointer', 'host', 'metadata_version', 'columns', 'mutation_pointer', 'min_unprocessed_insert_time', 'creator_info', 'is_active'] Sending request(xid=2): GetChildren(path='/clickhouse/tables/test/replicas/replica2/flags', watcher=None) Received response(xid=2): [] Sending request(xid=3): Delete(path='/clickhouse/tables/test/replicas/replica2/flags', version=-1) Received response(xid=3): True Sending request(xid=4): GetChildren(path='/clickhouse/tables/test/replicas/replica2/is_lost', watcher=None) Received response(xid=4): [] Sending request(xid=5): Delete(path='/clickhouse/tables/test/replicas/replica2/is_lost', version=-1) Received response(xid=5): True Sending request(xid=6): GetChildren(path='/clickhouse/tables/test/replicas/replica2/metadata', watcher=None) Received response(xid=6): [] Sending request(xid=7): Delete(path='/clickhouse/tables/test/replicas/replica2/metadata', version=-1) Received response(xid=7): True Sending request(xid=8): GetChildren(path='/clickhouse/tables/test/replicas/replica2/max_processed_insert_time', watcher=None) Received response(xid=8): [] Sending request(xid=9): Delete(path='/clickhouse/tables/test/replicas/replica2/max_processed_insert_time', version=-1) Received response(xid=9): True Sending request(xid=10): GetChildren(path='/clickhouse/tables/test/replicas/replica2/queue', watcher=None) Received response(xid=10): [] Sending request(xid=11): Delete(path='/clickhouse/tables/test/replicas/replica2/queue', version=-1) Received response(xid=11): True Sending request(xid=12): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts', watcher=None) Received response(xid=12): ['3_2_2_0', '4_4_4_0', '5_0_4_1', '3_4_4_0', '7_0_0_0', '3_3_3_0', '9_4_4_0', '6_1_1_0', '5_1_1_0', '7_3_3_0', '1_4_4_0', '1_3_3_0', '4_3_3_0', '9_0_0_0', '8_4_4_0', '9_1_1_0', '7_1_1_0', '4_0_4_1', '2_3_3_0', '4_2_2_0', '6_0_4_1', '9_2_2_0', '3_0_4_1', '1_2_2_0', '0_0_4_1', '6_3_3_0', '6_2_2_0', '0_4_4_0', '6_0_0_0', '4_0_0_0', '0_0_0_0', '1_1_1_0', '3_1_1_0', '5_0_0_0', '8_0_4_1', '5_3_3_0', '4_1_1_0', '2_2_2_0', '0_1_1_0', '9_3_3_0', '5_4_4_0', '8_1_1_0', '2_0_4_1', '2_0_0_0', '8_2_2_0', '8_3_3_0', '1_0_0_0', '2_1_1_0', '6_4_4_0', '0_2_2_0', '0_3_3_0', '2_4_4_0', '7_0_4_1', '7_4_4_0', '1_0_4_1', '7_2_2_0', '9_0_4_1', '8_0_0_0', '5_2_2_0', '3_0_0_0'] [gw4] PASSED test_s3_cluster/test.py::test_cluster_with_named_collection Sending request(xid=13): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/3_2_2_0', watcher=None) Received response(xid=13): [] Sending request(xid=14): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/3_2_2_0', version=-1) test_s3_cluster/test.py::test_count Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 Received response(xid=14): True Sending request(xid=15): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/4_4_4_0', watcher=None) Received response(xid=15): [] Sending request(xid=16): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/4_4_4_0', version=-1) Received response(xid=16): True Sending request(xid=17): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/5_0_4_1', watcher=None) Received response(xid=17): [] Sending request(xid=18): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/5_0_4_1', version=-1) Received response(xid=18): True Sending request(xid=19): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/3_4_4_0', watcher=None) Received response(xid=19): [] Sending request(xid=20): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/3_4_4_0', version=-1) Received response(xid=20): True Sending request(xid=21): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/7_0_0_0', watcher=None) Received response(xid=21): [] Sending request(xid=22): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/7_0_0_0', version=-1) Received response(xid=22): True Sending request(xid=23): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/3_3_3_0', watcher=None) Received response(xid=23): [] Sending request(xid=24): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/3_3_3_0', version=-1) Executing query DROP TABLE IF EXISTS mytable SYNC on node Received response(xid=24): True Sending request(xid=25): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/9_4_4_0', watcher=None) Received response(xid=25): [] Sending request(xid=26): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/9_4_4_0', version=-1) Received response(xid=26): True Sending request(xid=27): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/6_1_1_0', watcher=None) Received response(xid=27): [] Sending request(xid=28): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/6_1_1_0', version=-1) Executing query REVOKE ALL FROM B on instance Received response(xid=28): True Sending request(xid=29): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/5_1_1_0', watcher=None) Received response(xid=29): [] Sending request(xid=30): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/5_1_1_0', version=-1) Received response(xid=30): True Sending request(xid=31): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/7_3_3_0', watcher=None) Received response(xid=31): [] Sending request(xid=32): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/7_3_3_0', version=-1) Received response(xid=32): True Sending request(xid=33): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/1_4_4_0', watcher=None) Received response(xid=33): [] Sending request(xid=34): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/1_4_4_0', version=-1) Received response(xid=34): True Sending request(xid=35): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/1_3_3_0', watcher=None) Received response(xid=35): [] Sending request(xid=36): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/1_3_3_0', version=-1) Received response(xid=36): True Sending request(xid=37): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/4_3_3_0', watcher=None) Received response(xid=37): [] Sending request(xid=38): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/4_3_3_0', version=-1) Received response(xid=38): True Sending request(xid=39): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/9_0_0_0', watcher=None) Received response(xid=39): [] Sending request(xid=40): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/9_0_0_0', version=-1) Received response(xid=40): True Sending request(xid=41): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/8_4_4_0', watcher=None) Received response(xid=41): [] Sending request(xid=42): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/8_4_4_0', version=-1) Received response(xid=42): True http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Sending request(xid=43): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/9_1_1_0', watcher=None) Received response(xid=43): [] Sending request(xid=44): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/9_1_1_0', version=-1) Received response(xid=44): True Sending request(xid=45): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/7_1_1_0', watcher=None) Received response(xid=45): [] Sending request(xid=46): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/7_1_1_0', version=-1) Received response(xid=46): True Sending request(xid=47): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/4_0_4_1', watcher=None) Received response(xid=47): [] Sending request(xid=48): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/4_0_4_1', version=-1) Received response(xid=48): True Sending request(xid=49): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/2_3_3_0', watcher=None) Received response(xid=49): [] Sending request(xid=50): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/2_3_3_0', version=-1) Received response(xid=50): True Sending request(xid=51): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/4_2_2_0', watcher=None) Received response(xid=51): [] Sending request(xid=52): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/4_2_2_0', version=-1) Received response(xid=52): True Sending request(xid=53): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/6_0_4_1', watcher=None) Received response(xid=53): [] Sending request(xid=54): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/6_0_4_1', version=-1) Received response(xid=54): True Sending request(xid=55): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/9_2_2_0', watcher=None) Received response(xid=55): [] Sending request(xid=56): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/9_2_2_0', version=-1) Received response(xid=56): True Sending request(xid=57): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/3_0_4_1', watcher=None) Received response(xid=57): [] Sending request(xid=58): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/3_0_4_1', version=-1) Received response(xid=58): True Sending request(xid=59): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/1_2_2_0', watcher=None) Received response(xid=59): [] Sending request(xid=60): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/1_2_2_0', version=-1) Received response(xid=60): True Sending request(xid=61): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/0_0_4_1', watcher=None) Received response(xid=61): [] Sending request(xid=62): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/0_0_4_1', version=-1) Received response(xid=62): True Sending request(xid=63): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/6_3_3_0', watcher=None) Received response(xid=63): [] Sending request(xid=64): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/6_3_3_0', version=-1) Received response(xid=64): True Sending request(xid=65): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/6_2_2_0', watcher=None) Received response(xid=65): [] Sending request(xid=66): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/6_2_2_0', version=-1) Received response(xid=66): True Sending request(xid=67): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/0_4_4_0', watcher=None) Received response(xid=67): [] Sending request(xid=68): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/0_4_4_0', version=-1) Received response(xid=68): True Sending request(xid=69): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/6_0_0_0', watcher=None) Received response(xid=69): [] Sending request(xid=70): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/6_0_0_0', version=-1) Received response(xid=70): True Sending request(xid=71): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/4_0_0_0', watcher=None) Received response(xid=71): [] Sending request(xid=72): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/4_0_0_0', version=-1) Received response(xid=72): True Sending request(xid=73): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/0_0_0_0', watcher=None) Received response(xid=73): [] Sending request(xid=74): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/0_0_0_0', version=-1) Received response(xid=74): True Sending request(xid=75): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/1_1_1_0', watcher=None) Received response(xid=75): [] Sending request(xid=76): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/1_1_1_0', version=-1) Executing query SELECT count(*) from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 Received response(xid=76): True Sending request(xid=77): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/3_1_1_0', watcher=None) Received response(xid=77): [] Sending request(xid=78): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/3_1_1_0', version=-1) Received response(xid=78): True Sending request(xid=79): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/5_0_0_0', watcher=None) Received response(xid=79): [] Sending request(xid=80): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/5_0_0_0', version=-1) Received response(xid=80): True Sending request(xid=81): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/8_0_4_1', watcher=None) Received response(xid=81): [] Sending request(xid=82): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/8_0_4_1', version=-1) Received response(xid=82): True Sending request(xid=83): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/5_3_3_0', watcher=None) Received response(xid=83): [] Sending request(xid=84): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/5_3_3_0', version=-1) Received response(xid=84): True Sending request(xid=85): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/4_1_1_0', watcher=None) Received response(xid=85): [] Sending request(xid=86): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/4_1_1_0', version=-1) Received response(xid=86): True Sending request(xid=87): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/2_2_2_0', watcher=None) Received response(xid=87): [] Sending request(xid=88): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/2_2_2_0', version=-1) Received response(xid=88): True Executing query DROP TABLE IF EXISTS mymetrics SYNC on node Sending request(xid=89): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/0_1_1_0', watcher=None) Received response(xid=89): [] Sending request(xid=90): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/0_1_1_0', version=-1) Received response(xid=90): True Sending request(xid=91): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/9_3_3_0', watcher=None) Received response(xid=91): [] Sending request(xid=92): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/9_3_3_0', version=-1) Received response(xid=92): True Executing query SHOW GRANTS FOR B on instance Sending request(xid=93): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/5_4_4_0', watcher=None) Received response(xid=93): [] Sending request(xid=94): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/5_4_4_0', version=-1) Received response(xid=94): True Sending request(xid=95): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/8_1_1_0', watcher=None) Received response(xid=95): [] Sending request(xid=96): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/8_1_1_0', version=-1) Received response(xid=96): True Sending request(xid=97): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/2_0_4_1', watcher=None) Received response(xid=97): [] Sending request(xid=98): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/2_0_4_1', version=-1) Received response(xid=98): True Sending request(xid=99): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/2_0_0_0', watcher=None) Received response(xid=99): [] Sending request(xid=100): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/2_0_0_0', version=-1) Received response(xid=100): True Sending request(xid=101): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/8_2_2_0', watcher=None) Received response(xid=101): [] Sending request(xid=102): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/8_2_2_0', version=-1) Received response(xid=102): True Sending request(xid=103): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/8_3_3_0', watcher=None) Received response(xid=103): [] Sending request(xid=104): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/8_3_3_0', version=-1) Received response(xid=104): True Sending request(xid=105): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/1_0_0_0', watcher=None) Received response(xid=105): [] Sending request(xid=106): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/1_0_0_0', version=-1) Received response(xid=106): True Sending request(xid=107): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/2_1_1_0', watcher=None) Received response(xid=107): [] Sending request(xid=108): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/2_1_1_0', version=-1) Received response(xid=108): True Sending request(xid=109): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/6_4_4_0', watcher=None) Received response(xid=109): [] Sending request(xid=110): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/6_4_4_0', version=-1) Received response(xid=110): True Sending request(xid=111): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/0_2_2_0', watcher=None) Received response(xid=111): [] Sending request(xid=112): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/0_2_2_0', version=-1) Received response(xid=112): True Sending request(xid=113): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/0_3_3_0', watcher=None) Received response(xid=113): [] Sending request(xid=114): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/0_3_3_0', version=-1) Received response(xid=114): True Sending request(xid=115): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/2_4_4_0', watcher=None) Received response(xid=115): [] Sending request(xid=116): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/2_4_4_0', version=-1) Received response(xid=116): True Sending request(xid=117): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/7_0_4_1', watcher=None) Received response(xid=117): [] Sending request(xid=118): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/7_0_4_1', version=-1) Received response(xid=118): True Sending request(xid=119): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/7_4_4_0', watcher=None) Received response(xid=119): [] Sending request(xid=120): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/7_4_4_0', version=-1) Received response(xid=120): True Sending request(xid=121): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/1_0_4_1', watcher=None) Received response(xid=121): [] Sending request(xid=122): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/1_0_4_1', version=-1) Received response(xid=122): True Sending request(xid=123): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/7_2_2_0', watcher=None) Received response(xid=123): [] Sending request(xid=124): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/7_2_2_0', version=-1) Received response(xid=124): True Sending request(xid=125): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/9_0_4_1', watcher=None) Received response(xid=125): [] Sending request(xid=126): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/9_0_4_1', version=-1) Received response(xid=126): True Sending request(xid=127): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/8_0_0_0', watcher=None) Received response(xid=127): [] Sending request(xid=128): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/8_0_0_0', version=-1) Received response(xid=128): True Sending request(xid=129): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/5_2_2_0', watcher=None) Received response(xid=129): [] Sending request(xid=130): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/5_2_2_0', version=-1) Received response(xid=130): True Sending request(xid=131): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/3_0_0_0', watcher=None) Received response(xid=131): [] Sending request(xid=132): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/3_0_0_0', version=-1) Received response(xid=132): True Sending request(xid=133): Delete(path='/clickhouse/tables/test/replicas/replica2/parts', version=-1) Received response(xid=133): True Sending request(xid=134): GetChildren(path='/clickhouse/tables/test/replicas/replica2/log_pointer', watcher=None) Received response(xid=134): [] Sending request(xid=135): Delete(path='/clickhouse/tables/test/replicas/replica2/log_pointer', version=-1) Received response(xid=135): True Sending request(xid=136): GetChildren(path='/clickhouse/tables/test/replicas/replica2/host', watcher=None) Received response(xid=136): [] Sending request(xid=137): Delete(path='/clickhouse/tables/test/replicas/replica2/host', version=-1) Received response(xid=137): True Sending request(xid=138): GetChildren(path='/clickhouse/tables/test/replicas/replica2/metadata_version', watcher=None) Received response(xid=138): [] Sending request(xid=139): Delete(path='/clickhouse/tables/test/replicas/replica2/metadata_version', version=-1) http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Received response(xid=139): True Sending request(xid=140): GetChildren(path='/clickhouse/tables/test/replicas/replica2/columns', watcher=None) Received response(xid=140): [] Sending request(xid=141): Delete(path='/clickhouse/tables/test/replicas/replica2/columns', version=-1) Received response(xid=141): True Sending request(xid=142): GetChildren(path='/clickhouse/tables/test/replicas/replica2/mutation_pointer', watcher=None) Received response(xid=142): [] Sending request(xid=143): Delete(path='/clickhouse/tables/test/replicas/replica2/mutation_pointer', version=-1) Received response(xid=143): True Sending request(xid=144): GetChildren(path='/clickhouse/tables/test/replicas/replica2/min_unprocessed_insert_time', watcher=None) Received response(xid=144): [] Sending request(xid=145): Delete(path='/clickhouse/tables/test/replicas/replica2/min_unprocessed_insert_time', version=-1) Received response(xid=145): True Sending request(xid=146): GetChildren(path='/clickhouse/tables/test/replicas/replica2/creator_info', watcher=None) Received response(xid=146): [] Sending request(xid=147): Delete(path='/clickhouse/tables/test/replicas/replica2/creator_info', version=-1) Received response(xid=147): True Sending request(xid=148): GetChildren(path='/clickhouse/tables/test/replicas/replica2/is_active', watcher=None) Received response(xid=148): [] Sending request(xid=149): Delete(path='/clickhouse/tables/test/replicas/replica2/is_active', version=-1) test_prometheus_protocols/test.py::test_default Executing query CREATE TABLE prometheus ENGINE=TimeSeries on node Received response(xid=149): True Sending request(xid=150): Delete(path='/clickhouse/tables/test/replicas/replica2', version=-1) Executing query GRANT R1, R2 TO B on instance Received response(xid=150): True Sending request(xid=151): Exists(path='/clickhouse/tables/test/replicas/replica2', watcher=None) Executing query SYSTEM RESTORE REPLICA test on replica1 Executing query SHOW GRANTS FOR B on instance [gw4] PASSED test_s3_cluster/test.py::test_count Executing query SELECT count(*) from s3Cluster( '{default_cluster_macro}', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 test_s3_cluster/test.py::test_count_macro Starting new HTTP connection (1): 172.16.10.3:9090 Sending request(xid=152): GetChildren(path='/clickhouse/tables/test/replicas/replica1', watcher=None) http://172.16.10.3:9090 "GET /api/v1/query?query=up&time=1743566308.7599242 HTTP/1.1" 200 161 Received response(xid=152): ['flags', 'is_lost', 'metadata', 'max_processed_insert_time', 'queue', 'parts', 'log_pointer', 'host', 'metadata_version', 'columns', 'mutation_pointer', 'min_unprocessed_insert_time', 'creator_info', 'is_active'] Sending request(xid=153): GetChildren(path='/clickhouse/tables/test/replicas/replica1/flags', watcher=None) Received response(xid=153): [] Sending request(xid=154): Delete(path='/clickhouse/tables/test/replicas/replica1/flags', version=-1) Received response(xid=154): True Sending request(xid=155): GetChildren(path='/clickhouse/tables/test/replicas/replica1/is_lost', watcher=None) Received response(xid=155): [] Sending request(xid=156): Delete(path='/clickhouse/tables/test/replicas/replica1/is_lost', version=-1) Received response(xid=156): True Sending request(xid=157): GetChildren(path='/clickhouse/tables/test/replicas/replica1/metadata', watcher=None) Received response(xid=157): [] Sending request(xid=158): Delete(path='/clickhouse/tables/test/replicas/replica1/metadata', version=-1) Received response(xid=158): True Sending request(xid=159): GetChildren(path='/clickhouse/tables/test/replicas/replica1/max_processed_insert_time', watcher=None) Received response(xid=159): [] Sending request(xid=160): Delete(path='/clickhouse/tables/test/replicas/replica1/max_processed_insert_time', version=-1) Received response(xid=160): True Sending request(xid=161): GetChildren(path='/clickhouse/tables/test/replicas/replica1/queue', watcher=None) Received response(xid=161): [] Sending request(xid=162): Delete(path='/clickhouse/tables/test/replicas/replica1/queue', version=-1) Received response(xid=162): True Sending request(xid=163): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts', watcher=None) Received response(xid=163): ['5_0_4_1', '4_4_4_0', '3_2_2_0', '7_0_0_0', '3_4_4_0', '9_4_4_0', '3_3_3_0', '6_1_1_0', '1_4_4_0', '5_1_1_0', '7_3_3_0', '9_0_0_0', '1_3_3_0', '4_3_3_0', '8_4_4_0', '9_1_1_0', '7_1_1_0', '4_0_4_1', '4_2_2_0', '2_3_3_0', '6_0_4_1', '1_2_2_0', '9_2_2_0', '3_0_4_1', '6_3_3_0', '0_0_4_1', '6_2_2_0', '4_0_0_0', '6_0_0_0', '0_4_4_0', '0_0_0_0', '1_1_1_0', '3_1_1_0', '5_0_0_0', '5_3_3_0', '8_0_4_1', '4_1_1_0', '2_2_2_0', '9_3_3_0', '0_1_1_0', '8_1_1_0', '5_4_4_0', '2_0_4_1', '8_2_2_0', '2_0_0_0', '8_3_3_0', '0_2_2_0', '6_4_4_0', '7_0_4_1', '1_0_0_0', '7_4_4_0', '2_1_1_0', '0_3_3_0', '2_4_4_0', '1_0_4_1', '9_0_4_1', '7_2_2_0', '5_2_2_0', '8_0_0_0', '3_0_0_0'] Sending request(xid=164): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/5_0_4_1', watcher=None) Received response(xid=164): [] Sending request(xid=165): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/5_0_4_1', version=-1) Received response(xid=165): True Sending request(xid=166): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/4_4_4_0', watcher=None) Received response(xid=166): [] Sending request(xid=167): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/4_4_4_0', version=-1) Received response(xid=167): True Sending request(xid=168): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/3_2_2_0', watcher=None) Received response(xid=168): [] Sending request(xid=169): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/3_2_2_0', version=-1) Received response(xid=169): True Sending request(xid=170): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/7_0_0_0', watcher=None) Received response(xid=170): [] Sending request(xid=171): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/7_0_0_0', version=-1) Received response(xid=171): True Sending request(xid=172): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/3_4_4_0', watcher=None) http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Received response(xid=172): [] Sending request(xid=173): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/3_4_4_0', version=-1) Received response(xid=173): True Sending request(xid=174): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/9_4_4_0', watcher=None) Received response(xid=174): [] Sending request(xid=175): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/9_4_4_0', version=-1) Received response(xid=175): True Sending request(xid=176): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/3_3_3_0', watcher=None) Received response(xid=176): [] Sending request(xid=177): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/3_3_3_0', version=-1) Received response(xid=177): True Executing query REVOKE ALL FROM B on instance Sending request(xid=178): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/6_1_1_0', watcher=None) Received response(xid=178): [] Sending request(xid=179): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/6_1_1_0', version=-1) Received response(xid=179): True Sending request(xid=180): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/1_4_4_0', watcher=None) Received response(xid=180): [] Sending request(xid=181): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/1_4_4_0', version=-1) Received response(xid=181): True Sending request(xid=182): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/5_1_1_0', watcher=None) Received response(xid=182): [] Sending request(xid=183): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/5_1_1_0', version=-1) Received response(xid=183): True Sending request(xid=184): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/7_3_3_0', watcher=None) Received response(xid=184): [] Sending request(xid=185): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/7_3_3_0', version=-1) Received response(xid=185): True Sending request(xid=186): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/9_0_0_0', watcher=None) Received response(xid=186): [] Sending request(xid=187): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/9_0_0_0', version=-1) Received response(xid=187): True Sending request(xid=188): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/1_3_3_0', watcher=None) Received response(xid=188): [] Sending request(xid=189): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/1_3_3_0', version=-1) Received response(xid=189): True Sending request(xid=190): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/4_3_3_0', watcher=None) Received response(xid=190): [] Sending request(xid=191): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/4_3_3_0', version=-1) Received response(xid=191): True Sending request(xid=192): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/8_4_4_0', watcher=None) Received response(xid=192): [] Sending request(xid=193): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/8_4_4_0', version=-1) Received response(xid=193): True Sending request(xid=194): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/9_1_1_0', watcher=None) Received response(xid=194): [] Sending request(xid=195): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/9_1_1_0', version=-1) Received response(xid=195): True Sending request(xid=196): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/7_1_1_0', watcher=None) Received response(xid=196): [] Sending request(xid=197): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/7_1_1_0', version=-1) Received response(xid=197): True Sending request(xid=198): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/4_0_4_1', watcher=None) Received response(xid=198): [] Sending request(xid=199): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/4_0_4_1', version=-1) Received response(xid=199): True Sending request(xid=200): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/4_2_2_0', watcher=None) Received response(xid=200): [] Sending request(xid=201): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/4_2_2_0', version=-1) Received response(xid=201): True Sending request(xid=202): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/2_3_3_0', watcher=None) Received response(xid=202): [] Sending request(xid=203): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/2_3_3_0', version=-1) Received response(xid=203): True Sending request(xid=204): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/6_0_4_1', watcher=None) Received response(xid=204): [] Sending request(xid=205): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/6_0_4_1', version=-1) Received response(xid=205): True Sending request(xid=206): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/1_2_2_0', watcher=None) Received response(xid=206): [] Sending request(xid=207): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/1_2_2_0', version=-1) Received response(xid=207): True Sending request(xid=208): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/9_2_2_0', watcher=None) Received response(xid=208): [] Sending request(xid=209): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/9_2_2_0', version=-1) Received response(xid=209): True Sending request(xid=210): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/3_0_4_1', watcher=None) Received response(xid=210): [] Sending request(xid=211): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/3_0_4_1', version=-1) Received response(xid=211): True Sending request(xid=212): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/6_3_3_0', watcher=None) Received response(xid=212): [] Sending request(xid=213): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/6_3_3_0', version=-1) Received response(xid=213): True Sending request(xid=214): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/0_0_4_1', watcher=None) Received response(xid=214): [] Sending request(xid=215): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/0_0_4_1', version=-1) Received response(xid=215): True Sending request(xid=216): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/6_2_2_0', watcher=None) Received response(xid=216): [] Sending request(xid=217): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/6_2_2_0', version=-1) Received response(xid=217): True Sending request(xid=218): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/4_0_0_0', watcher=None) Received response(xid=218): [] Sending request(xid=219): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/4_0_0_0', version=-1) Received response(xid=219): True Sending request(xid=220): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/6_0_0_0', watcher=None) Received response(xid=220): [] Sending request(xid=221): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/6_0_0_0', version=-1) Received response(xid=221): True Sending request(xid=222): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/0_4_4_0', watcher=None) Received response(xid=222): [] Sending request(xid=223): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/0_4_4_0', version=-1) Received response(xid=223): True Sending request(xid=224): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/0_0_0_0', watcher=None) Received response(xid=224): [] Sending request(xid=225): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/0_0_0_0', version=-1) Received response(xid=225): True Sending request(xid=226): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/1_1_1_0', watcher=None) Executing query SELECT count(*) from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 Received response(xid=226): [] Sending request(xid=227): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/1_1_1_0', version=-1) Received response(xid=227): True Sending request(xid=228): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/3_1_1_0', watcher=None) Received response(xid=228): [] Sending request(xid=229): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/3_1_1_0', version=-1) Received response(xid=229): True Sending request(xid=230): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/5_0_0_0', watcher=None) Received response(xid=230): [] Sending request(xid=231): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/5_0_0_0', version=-1) Received response(xid=231): True Sending request(xid=232): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/5_3_3_0', watcher=None) Received response(xid=232): [] Sending request(xid=233): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/5_3_3_0', version=-1) Received response(xid=233): True Sending request(xid=234): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/8_0_4_1', watcher=None) Received response(xid=234): [] Sending request(xid=235): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/8_0_4_1', version=-1) Received response(xid=235): True Sending request(xid=236): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/4_1_1_0', watcher=None) Executing query SHOW GRANTS FOR B on instance Received response(xid=236): [] Sending request(xid=237): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/4_1_1_0', version=-1) Received response(xid=237): True Sending request(xid=238): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/2_2_2_0', watcher=None) Received response(xid=238): [] Sending request(xid=239): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/2_2_2_0', version=-1) Received response(xid=239): True Sending request(xid=240): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/9_3_3_0', watcher=None) Received response(xid=240): [] Sending request(xid=241): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/9_3_3_0', version=-1) Received response(xid=241): True Sending request(xid=242): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/0_1_1_0', watcher=None) Received response(xid=242): [] Sending request(xid=243): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/0_1_1_0', version=-1) Received response(xid=243): True Sending request(xid=244): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/8_1_1_0', watcher=None) Received response(xid=244): [] Sending request(xid=245): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/8_1_1_0', version=-1) Received response(xid=245): True Sending request(xid=246): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/5_4_4_0', watcher=None) Received response(xid=246): [] Sending request(xid=247): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/5_4_4_0', version=-1) Received response(xid=247): True Sending request(xid=248): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/2_0_4_1', watcher=None) Received response(xid=248): [] Sending request(xid=249): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/2_0_4_1', version=-1) Received response(xid=249): True Sending request(xid=250): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/8_2_2_0', watcher=None) Received response(xid=250): [] Sending request(xid=251): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/8_2_2_0', version=-1) Received response(xid=251): True Sending request(xid=252): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/2_0_0_0', watcher=None) Received response(xid=252): [] Sending request(xid=253): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/2_0_0_0', version=-1) Received response(xid=253): True Sending request(xid=254): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/8_3_3_0', watcher=None) Received response(xid=254): [] Sending request(xid=255): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/8_3_3_0', version=-1) Received response(xid=255): True Sending request(xid=256): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/0_2_2_0', watcher=None) Received response(xid=256): [] Sending request(xid=257): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/0_2_2_0', version=-1) Received response(xid=257): True Sending request(xid=258): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/6_4_4_0', watcher=None) Received response(xid=258): [] Sending request(xid=259): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/6_4_4_0', version=-1) Received response(xid=259): True Sending request(xid=260): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/7_0_4_1', watcher=None) Received response(xid=260): [] Sending request(xid=261): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/7_0_4_1', version=-1) Received response(xid=261): True Sending request(xid=262): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/1_0_0_0', watcher=None) Received response(xid=262): [] Sending request(xid=263): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/1_0_0_0', version=-1) Received response(xid=263): True Sending request(xid=264): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/7_4_4_0', watcher=None) Received response(xid=264): [] Sending request(xid=265): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/7_4_4_0', version=-1) http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Received response(xid=265): True Sending request(xid=266): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/2_1_1_0', watcher=None) Received response(xid=266): [] Sending request(xid=267): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/2_1_1_0', version=-1) Received response(xid=267): True Sending request(xid=268): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/0_3_3_0', watcher=None) Received response(xid=268): [] Sending request(xid=269): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/0_3_3_0', version=-1) Received response(xid=269): True Sending request(xid=270): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/2_4_4_0', watcher=None) Received response(xid=270): [] Sending request(xid=271): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/2_4_4_0', version=-1) Received response(xid=271): True Sending request(xid=272): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/1_0_4_1', watcher=None) Received response(xid=272): [] Sending request(xid=273): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/1_0_4_1', version=-1) Received response(xid=273): True Sending request(xid=274): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/9_0_4_1', watcher=None) Received response(xid=274): [] Sending request(xid=275): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/9_0_4_1', version=-1) Received response(xid=275): True Sending request(xid=276): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/7_2_2_0', watcher=None) Received response(xid=276): [] Sending request(xid=277): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/7_2_2_0', version=-1) Received response(xid=277): True Sending request(xid=278): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/5_2_2_0', watcher=None) Received response(xid=278): [] Sending request(xid=279): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/5_2_2_0', version=-1) Received response(xid=279): True Sending request(xid=280): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/8_0_0_0', watcher=None) Received response(xid=280): [] Sending request(xid=281): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/8_0_0_0', version=-1) Received response(xid=281): True Sending request(xid=282): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/3_0_0_0', watcher=None) Received response(xid=282): [] Sending request(xid=283): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/3_0_0_0', version=-1) Received response(xid=283): True Sending request(xid=284): Delete(path='/clickhouse/tables/test/replicas/replica1/parts', version=-1) Received response(xid=284): True Sending request(xid=285): GetChildren(path='/clickhouse/tables/test/replicas/replica1/log_pointer', watcher=None) Received response(xid=285): [] Sending request(xid=286): Delete(path='/clickhouse/tables/test/replicas/replica1/log_pointer', version=-1) Received response(xid=286): True Sending request(xid=287): GetChildren(path='/clickhouse/tables/test/replicas/replica1/host', watcher=None) Received response(xid=287): [] Sending request(xid=288): Delete(path='/clickhouse/tables/test/replicas/replica1/host', version=-1) Received response(xid=288): True Sending request(xid=289): GetChildren(path='/clickhouse/tables/test/replicas/replica1/metadata_version', watcher=None) Received response(xid=289): [] Sending request(xid=290): Delete(path='/clickhouse/tables/test/replicas/replica1/metadata_version', version=-1) Received response(xid=290): True Sending request(xid=291): GetChildren(path='/clickhouse/tables/test/replicas/replica1/columns', watcher=None) Received response(xid=291): [] Sending request(xid=292): Delete(path='/clickhouse/tables/test/replicas/replica1/columns', version=-1) Received response(xid=292): True Executing query REVOKE ALL EXCEPT R2 FROM B on instance Sending request(xid=293): GetChildren(path='/clickhouse/tables/test/replicas/replica1/mutation_pointer', watcher=None) Received response(xid=293): [] Sending request(xid=294): Delete(path='/clickhouse/tables/test/replicas/replica1/mutation_pointer', version=-1) Received response(xid=294): True Sending request(xid=295): GetChildren(path='/clickhouse/tables/test/replicas/replica1/min_unprocessed_insert_time', watcher=None) Executing query SELECT id FROM test_table order by id on node5 Received response(xid=295): [] Sending request(xid=296): Delete(path='/clickhouse/tables/test/replicas/replica1/min_unprocessed_insert_time', version=-1) Received response(xid=296): True Sending request(xid=297): GetChildren(path='/clickhouse/tables/test/replicas/replica1/creator_info', watcher=None) Received response(xid=297): [] Sending request(xid=298): Delete(path='/clickhouse/tables/test/replicas/replica1/creator_info', version=-1) Received response(xid=298): True Sending request(xid=299): GetChildren(path='/clickhouse/tables/test/replicas/replica1/is_active', watcher=None) Received response(xid=299): [] Sending request(xid=300): Delete(path='/clickhouse/tables/test/replicas/replica1/is_active', version=-1) Received response(xid=300): True Sending request(xid=301): Delete(path='/clickhouse/tables/test/replicas/replica1', version=-1) Received response(xid=301): True Sending request(xid=302): Exists(path='/clickhouse/tables/test/replicas/replica1', watcher=None) Executing query SYSTEM RESTART REPLICA test on replica1 Executing query SHOW GRANTS FOR B on instance [gw4] PASSED test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated Executing query DROP TABLE IF EXISTS insert_select_replicated_local ON CLUSTER 'first_shard' SYNC; on s0_0_0 Executing query SELECT id FROM test_table order by id on node6 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SYSTEM RESTORE REPLICA test on replica1 Executing query GRANT R2 TO A WITH ADMIN OPTION on instance Executing query insert into test_table values ('2017-06-21', 222, 1) on node6 Executing query REVOKE ALL FROM B on instance http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SHOW GRANTS FOR B on instance Executing query SYSTEM RESTART REPLICA test on replica2 Executing query CREATE TABLE insert_select_replicated_local ON CLUSTER 'first_shard' (a String, b UInt64) ENGINE=ReplicatedMergeTree('/clickhouse/tables/{shard}/insert_select_with_replicated', '{replica}') ORDER BY (a, b); on s0_0_0 Executing query GRANT R1, R2 TO B on instance Executing query SYSTEM RESTORE REPLICA test on replica2 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SHOW GRANTS FOR B on instance run container_id:roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SYSTEM STOP FETCHES; on s0_0_0 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SELECT sum(n), count() FROM test on replica1 Stdout:9 Executing query SYSTEM STOP MERGES; on s0_0_0 Executing query REVOKE ALL FROM B on instance Executing query SELECT sum(n), count() FROM test on replica2 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SYSTEM STOP FETCHES; on s0_0_1 Executing query SHOW GRANTS FOR B on instance Executing query SELECT sum(n), count() FROM test on replica3 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SYSTEM STOP MERGES; on s0_0_1 Executing query DROP USER IF EXISTS A, B on instance [gw0] PASSED test_role/test.py::test_revoke_requires_admin_option Executing query INSERT INTO test SELECT number + 1000 FROM numbers(1000) on replica1 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query INSERT INTO insert_select_replicated_local SELECT * FROM s3Cluster( 'first_shard', 'http://minio1:9001/root/data/generated/*.csv', 'minio', 'minio123', 'CSV','a String, b UInt64' ) SETTINGS parallel_distributed_insert_select=1; on s0_0_0 Executing query DROP ROLE IF EXISTS R1, R2, R3, R4 on instance Stderr: Container roottestreloadingsettingsfromusersxml-gw5-node-1 Stopping Stderr: Container roottestreloadingsettingsfromusersxml-gw5-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/_instances-0-gw5/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/_instances-0-gw5/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/_instances-0-gw5/.env --project-name roottestreloadingsettingsfromusersxml-gw5 --file /ClickHouse/tests/integration/test_reloading_settings_from_users_xml/_instances-0-gw5/node/docker-compose.yml down --volumes] http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SYSTEM SYNC REPLICA test on replica2 Executing query CREATE ROLE rre on instance test_role/test.py::test_role_expiration[False] Starting new HTTP connection (1): 172.16.10.2:9091 http://172.16.10.2:9091 "GET /api/v1/query?query=up&time=1743566308.7599242 HTTP/1.1" 200 87 Starting new HTTP connection (1): 172.16.10.3:9090 Executing query SYSTEM SYNC REPLICA test on replica3 http://172.16.10.3:9090 "GET /api/v1/query?query=up&time=1743566308.7599242 HTTP/1.1" 200 161 Executing query CREATE USER ure DEFAULT ROLE rre on instance http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SYSTEM FLUSH LOGS; on s0_0_0 Executing query CREATE TABLE table1 (id Int) Engine=Log on instance Executing query SELECT sum(n), count() FROM test on replica1 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query CREATE TABLE table2 (id Int) Engine=Log on instance Executing query INSERT INTO table1 VALUES (1) on instance Executing query SYSTEM FLUSH LOGS; on s0_0_1 Executing query SELECT sum(n), count() FROM test on replica2 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query INSERT INTO table2 VALUES (2) on instance Executing query GRANT SELECT ON table1 TO rre on instance Executing query SELECT id FROM test_table order by id on node5 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SELECT sum(n), count() FROM test on replica3 Executing query SELECT count(*) FROM system.query_log WHERE not is_initial_query and query ilike '%s3Cluster%'; on s0_0_1 Executing query SELECT * FROM table1 on instance Executing query SELECT id FROM test_table order by id on node6 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SYSTEM RESTORE REPLICA test on replica1 Stderr: Container roottestreloadingsettingsfromusersxml-gw5-node-1 Stopping Stderr: Container roottestreloadingsettingsfromusersxml-gw5-node-1 Stopped Stderr: Container roottestreloadingsettingsfromusersxml-gw5-node-1 Removing Stderr: Container roottestreloadingsettingsfromusersxml-gw5-node-1 Removed Stderr: Network roottestreloadingsettingsfromusersxml-gw5_default Removing Stderr: Network roottestreloadingsettingsfromusersxml-gw5_default Removed Cleanup called Executing query SELECT * FROM table2 on instance run container_id:roottestreplicationcredentials-gw8-node5-1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n 9009\n \n admin\n 222\n \n root\n 111\n \n \n aaa\n 333\n \n \n \n ' > /etc/clickhouse-server/config.d/credentials1.xml"] Command:[docker exec roottestreplicationcredentials-gw8-node5-1 bash -c echo ' 9009 admin 222 root 111 aaa 333 ' > /etc/clickhouse-server/config.d/credentials1.xml] Docker networks for project roottestreloadingsettingsfromusersxml-gw5 are NETWORK ID NAME DRIVER SCOPE Executing query SELECT count(*) FROM insert_select_replicated_local; on s0_0_1 Docker containers for project roottestreloadingsettingsfromusersxml-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreloadingsettingsfromusersxml-gw5 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreloadingsettingsfromusersxml-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Executing query SYSTEM RELOAD CONFIG on node5 Executing query SYSTEM RESTORE REPLICA test on replica2 Unstopped containers: {} No running containers for project: roottestreloadingsettingsfromusersxml-gw5 Trying to prune unused networks... run container_id:roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Trying to prune unused images... Command:[docker image prune -f] http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS insert_select_replicated_local ON CLUSTER 'first_shard' SYNC; on s0_0_0 Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query INSERT INTO test_table values('2017-06-21', 333, 1) on node5 Stdout:8 Command:[docker volume prune -f] Stdout:9 Executing query SYSTEM RESTORE REPLICA test on replica3 Stdout:Total reclaimed space: 0B Volumes pruned: 8 test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke Running tests in /ClickHouse/tests/integration/test_refreshable_mv/test.py Cluster start called. is_up=False Docker networks for project roottestrefreshablemv-gw5 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrefreshablemv-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query SYSTEM SYNC REPLICA test_table on node6 Docker volumes for project roottestrefreshablemv-gw5 are DRIVER VOLUME NAME Cleanup called http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None [gw9] PASSED test_restore_replica/test.py::test_restore_replica_alive_replicas Executing query SYSTEM RESTORE REPLICA i_dont_exist_42 on replica1 test_restore_replica/test.py::test_restore_replica_invalid_tables Docker networks for project roottestrefreshablemv-gw5 are NETWORK ID NAME DRIVER SCOPE [gw4] PASSED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/.env --project-name roottests3cluster-gw4 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_1_0/docker-compose.yml stop --timeout 20] Docker containers for project roottestrefreshablemv-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrefreshablemv-gw5 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrefreshablemv-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Executing query SELECT id FROM test_table order by id on node6 Unstopped containers: {} No running containers for project: roottestrefreshablemv-gw5 Trying to prune unused networks... Executing query SYSTEM RESTORE REPLICA no_db.i_dont_exist_42 on replica1 Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query SYSTEM RESTORE REPLICA system.numbers on replica1 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Stdout:8 Command:[docker volume prune -f] [gw8] PASSED test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials Running tests in /ClickHouse/tests/integration/test_replication_credentials/test.py Instance directory already exists. Did you call cluster.start() for second time? Cluster start called. is_up=True Stdout:Total reclaimed space: 0B Volumes pruned: 8 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_refreshable_mv/configs/config.xml'] to /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/database Setup logs dir /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_refreshable_mv/configs/config.xml'] to /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node2/database Setup logs dir /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Docker networks for project roottestreplicationcredentials-gw8 are NETWORK ID NAME DRIVER SCOPE 21b6fb026e24 roottestreplicationcredentials-gw8_default bridge local http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/.env --project-name roottestrefreshablemv-gw5 --file /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node2/docker-compose.yml pull] Docker containers for project roottestreplicationcredentials-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 325b4b255078 altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 8 seconds ago Up 7 seconds roottestreplicationcredentials-gw8-node6-1 29e73551c6ee altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 8 seconds ago Up 7 seconds roottestreplicationcredentials-gw8-node2-1 9ce4fa2057ea altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 8 seconds ago Up 7 seconds roottestreplicationcredentials-gw8-node1-1 921d2bc677a7 altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 8 seconds ago Up 7 seconds roottestreplicationcredentials-gw8-node7-1 839c507122d3 altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 8 seconds ago Up 7 seconds roottestreplicationcredentials-gw8-node3-1 b734901e175d altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 8 seconds ago Up 7 seconds roottestreplicationcredentials-gw8-node4-1 460fc5dd7dd0 altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 8 seconds ago Up 7 seconds roottestreplicationcredentials-gw8-node8-1 ea69f459f9cc altinityinfra/integration-test:8b2301119731 "clickhouse server -…" 8 seconds ago Up 7 seconds roottestreplicationcredentials-gw8-node5-1 6f1d895347f4 altinityinfra/integration-test:8b2301119731 "clickhouse keeper -…" 12 seconds ago Up 12 seconds roottestreplicationcredentials-gw8-zoo1-1 a1852004e00d altinityinfra/integration-test:8b2301119731 "clickhouse keeper -…" 12 seconds ago Up 12 seconds roottestreplicationcredentials-gw8-zoo3-1 2467e2a28494 altinityinfra/integration-test:8b2301119731 "clickhouse keeper -…" 12 seconds ago Up 12 seconds roottestreplicationcredentials-gw8-zoo2-1 get_instance_ip instance_name=zoo1 [gw9] PASSED test_restore_replica/test.py::test_restore_replica_invalid_tables test_restore_replica/test.py::test_restore_replica_parallel http://localhost:None "GET /v1.46/containers/roottestrestorereplica-gw9-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.5.3, port:2181, use_ssl:False Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Executing query DROP TABLE IF EXISTS test SYNC on replica1 Docker volumes for project roottestreplicationcredentials-gw8 are DRIVER VOLUME NAME Executing query CREATE DATABASE test; CREATE TABLE test_table(date Date, id UInt32, dummy UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test2/replicated', 'node3') PARTITION BY toYYYYMM(date) ORDER BY id; on node3 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query DROP TABLE IF EXISTS test SYNC on replica2 Executing query CREATE DATABASE test; CREATE TABLE test_table(date Date, id UInt32, dummy UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test2/replicated', 'node4') PARTITION BY toYYYYMM(date) ORDER BY id; on node4 Executing query insert into test_table values ('2017-06-18', 111, 0) on node3 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Starting new HTTP connection (1): 172.16.10.2:9091 Executing query DROP TABLE IF EXISTS test SYNC on replica3 http://172.16.10.2:9091 "GET /api/v1/query?query=up&time=1743566308.7599242 HTTP/1.1" 200 87 Starting new HTTP connection (1): 172.16.10.3:9090 http://172.16.10.3:9090 "GET /api/v1/query?query=up&time=1743566308.7599242 HTTP/1.1" 200 161 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query CREATE TABLE test(n UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/', 'replica1') ORDER BY n PARTITION BY n % 10; on replica1 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query CREATE TABLE test(n UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/', 'replica2') ORDER BY n PARTITION BY n % 10; on replica2 Executing query CREATE TABLE test(n UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/', 'replica3') ORDER BY n PARTITION BY n % 10; on replica3 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SELECT sum(n), count() FROM test on replica1 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SELECT sum(n), count() FROM test on replica2 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SELECT sum(n), count() FROM test on replica3 Executing query INSERT INTO test SELECT number + 0 FROM numbers(200) on replica1 run container_id:roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None run container_id:roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None http://localhost:None "POST /v1.46/containers/roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/4e1444a652b7ba87a6459c97d1fe2b694ca7fd224e939c3d9ea07b123578686e/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/4e1444a652b7ba87a6459c97d1fe2b694ca7fd224e939c3d9ea07b123578686e/json HTTP/1.1" 200 586 Executing query INSERT INTO test SELECT number + 200 FROM numbers(200) on replica1 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query INSERT INTO test SELECT number + 400 FROM numbers(200) on replica1 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query INSERT INTO test SELECT number + 600 FROM numbers(200) on replica1 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Starting new HTTP connection (1): 172.16.10.2:9091 Executing query SELECT id FROM test_table order by id on node3 http://172.16.10.2:9091 "GET /api/v1/query?query=up&time=1743566308.7599242 HTTP/1.1" 200 87 Starting new HTTP connection (1): 172.16.10.3:9090 http://172.16.10.3:9090 "GET /api/v1/query?query=up&time=1743566308.7599242 HTTP/1.1" 200 161 Executing query INSERT INTO test SELECT number + 800 FROM numbers(200) on replica1 Executing query SELECT id FROM test_table order by id on node4 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query insert into test_table values ('2017-06-19', 222, 1) on node4 Executing query SELECT sum(n), count() FROM test on replica1 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SELECT sum(n), count() FROM test on replica2 http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Executing query SELECT sum(n), count() FROM test on replica3 Sending request(xid=1): GetChildren(path='/clickhouse/tables/test', watcher=None) Received response(xid=1): ['blocks', 'part_moves_shard', 'async_blocks', 'log', 'replicas', 'alter_partition_version', 'leader_election', 'nonincrement_block_numbers', 'columns', 'pinned_part_uuids', 'block_numbers', 'mutations', 'temp', 'metadata', 'table_shared_id', 'quorum', 'lost_part_count'] Sending request(xid=2): GetChildren(path='/clickhouse/tables/test/blocks', watcher=None) Received response(xid=2): ['2_6842583636263644984_1982368210150799703', '6_14547442715190156624_1923972885002714920', '1_3239313922373837775_5565726551507660922', '2_8249225827185276881_2152826876730964536', '0_17476564874868548141_3454619666306521731', '0_8859342928210992409_11142976061697196078', '5_922816994910750113_7255101858939955030', '2_11536736153270116910_1741442162483686209', '0_4862134225373959152_16429649839201181738', '5_15633417433839596724_9532864273223696209', '9_1631987988450216801_4573880638144314625', '7_11228658654694027175_11036255538163384156', '4_1419910534893064631_10433669149697454996', '1_4223207937087353094_3040165143523075591', '8_4786891139465234359_16213969636893932635', '8_8149985435757468920_3110907703747737480', '8_5981173013080451276_15075524987330599189', '0_5823239430936010074_8336879650588380094', '9_2397390479047600166_1844210758128429459', '6_11800883215831824964_11063077091526655281', '4_7991306156949843873_10814474702301815977', '3_8943159229929697858_7063506240022641846', '9_2991137436629072923_1112121919112387392', '9_18111834145313106032_10223514217035097662', '5_11651285249085125834_14071769460747733447', '1_14567090541664558819_7702172552992592184', '7_193297420037564258_6066394564520291477', '1_8384938112033624258_12866389235029191354', '2_15871863537629735500_5393938433557777398', '2_7496753474834706969_10920641722843493311', '4_14956749016352276909_1845324259893722407', '9_4788800582724720834_14101928124621074680', '5_1736488526884124433_3887218309880210852', '7_9842665926788715729_16140505362770629564', '3_4110679356907143331_11035046055657096343', '4_2753186338986832608_776212573121335180', '3_2034507573975896007_4327225584940923981', '6_4626774046435431605_9754063117557495576', '3_13068591570044714032_5490360431194945326', '6_2088817551325874741_7861047104118916143', '8_12467143234983360370_14001134191946365082', '6_2855772191332027362_138014351105121520', '1_8011844867736689895_6905109647580634077', '3_16275396704443078712_13742690842691968439', '7_1015459668150584802_18117679876215610461', '4_10075552373073406857_12628533709889697856', '5_4397825682830605283_16079438150578917708', '0_9812559225968656149_12943158647658551309', '7_5315114371746958400_11285814637413696130', '8_4147808760916336577_11325747988160469001'] Sending request(xid=3): GetChildren(path='/clickhouse/tables/test/blocks/2_6842583636263644984_1982368210150799703', watcher=None) Received response(xid=3): [] Sending request(xid=4): Delete(path='/clickhouse/tables/test/blocks/2_6842583636263644984_1982368210150799703', version=-1) Received response(xid=4): True Sending request(xid=5): GetChildren(path='/clickhouse/tables/test/blocks/6_14547442715190156624_1923972885002714920', watcher=None) Received response(xid=5): [] Sending request(xid=6): Delete(path='/clickhouse/tables/test/blocks/6_14547442715190156624_1923972885002714920', version=-1) Received response(xid=6): True Sending request(xid=7): GetChildren(path='/clickhouse/tables/test/blocks/1_3239313922373837775_5565726551507660922', watcher=None) Received response(xid=7): [] Sending request(xid=8): Delete(path='/clickhouse/tables/test/blocks/1_3239313922373837775_5565726551507660922', version=-1) Received response(xid=8): True Sending request(xid=9): GetChildren(path='/clickhouse/tables/test/blocks/2_8249225827185276881_2152826876730964536', watcher=None) Received response(xid=9): [] Sending request(xid=10): Delete(path='/clickhouse/tables/test/blocks/2_8249225827185276881_2152826876730964536', version=-1) Received response(xid=10): True Sending request(xid=11): GetChildren(path='/clickhouse/tables/test/blocks/0_17476564874868548141_3454619666306521731', watcher=None) Received response(xid=11): [] Sending request(xid=12): Delete(path='/clickhouse/tables/test/blocks/0_17476564874868548141_3454619666306521731', version=-1) Received response(xid=12): True Sending request(xid=13): GetChildren(path='/clickhouse/tables/test/blocks/0_8859342928210992409_11142976061697196078', watcher=None) Received response(xid=13): [] Sending request(xid=14): Delete(path='/clickhouse/tables/test/blocks/0_8859342928210992409_11142976061697196078', version=-1) Received response(xid=14): True Sending request(xid=15): GetChildren(path='/clickhouse/tables/test/blocks/5_922816994910750113_7255101858939955030', watcher=None) Received response(xid=15): [] http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Sending request(xid=16): Delete(path='/clickhouse/tables/test/blocks/5_922816994910750113_7255101858939955030', version=-1) Received response(xid=16): True Sending request(xid=17): GetChildren(path='/clickhouse/tables/test/blocks/2_11536736153270116910_1741442162483686209', watcher=None) Received response(xid=17): [] Sending request(xid=18): Delete(path='/clickhouse/tables/test/blocks/2_11536736153270116910_1741442162483686209', version=-1) Received response(xid=18): True Sending request(xid=19): GetChildren(path='/clickhouse/tables/test/blocks/0_4862134225373959152_16429649839201181738', watcher=None) Received response(xid=19): [] Sending request(xid=20): Delete(path='/clickhouse/tables/test/blocks/0_4862134225373959152_16429649839201181738', version=-1) Received response(xid=20): True Sending request(xid=21): GetChildren(path='/clickhouse/tables/test/blocks/5_15633417433839596724_9532864273223696209', watcher=None) Received response(xid=21): [] Sending request(xid=22): Delete(path='/clickhouse/tables/test/blocks/5_15633417433839596724_9532864273223696209', version=-1) Received response(xid=22): True Sending request(xid=23): GetChildren(path='/clickhouse/tables/test/blocks/9_1631987988450216801_4573880638144314625', watcher=None) Received response(xid=23): [] Sending request(xid=24): Delete(path='/clickhouse/tables/test/blocks/9_1631987988450216801_4573880638144314625', version=-1) Received response(xid=24): True Sending request(xid=25): GetChildren(path='/clickhouse/tables/test/blocks/7_11228658654694027175_11036255538163384156', watcher=None) Received response(xid=25): [] Sending request(xid=26): Delete(path='/clickhouse/tables/test/blocks/7_11228658654694027175_11036255538163384156', version=-1) Received response(xid=26): True Sending request(xid=27): GetChildren(path='/clickhouse/tables/test/blocks/4_1419910534893064631_10433669149697454996', watcher=None) Received response(xid=27): [] Sending request(xid=28): Delete(path='/clickhouse/tables/test/blocks/4_1419910534893064631_10433669149697454996', version=-1) Received response(xid=28): True Sending request(xid=29): GetChildren(path='/clickhouse/tables/test/blocks/1_4223207937087353094_3040165143523075591', watcher=None) Received response(xid=29): [] Sending request(xid=30): Delete(path='/clickhouse/tables/test/blocks/1_4223207937087353094_3040165143523075591', version=-1) Received response(xid=30): True Sending request(xid=31): GetChildren(path='/clickhouse/tables/test/blocks/8_4786891139465234359_16213969636893932635', watcher=None) Received response(xid=31): [] Sending request(xid=32): Delete(path='/clickhouse/tables/test/blocks/8_4786891139465234359_16213969636893932635', version=-1) Received response(xid=32): True Sending request(xid=33): GetChildren(path='/clickhouse/tables/test/blocks/8_8149985435757468920_3110907703747737480', watcher=None) Received response(xid=33): [] Sending request(xid=34): Delete(path='/clickhouse/tables/test/blocks/8_8149985435757468920_3110907703747737480', version=-1) Received response(xid=34): True Sending request(xid=35): GetChildren(path='/clickhouse/tables/test/blocks/8_5981173013080451276_15075524987330599189', watcher=None) Received response(xid=35): [] Sending request(xid=36): Delete(path='/clickhouse/tables/test/blocks/8_5981173013080451276_15075524987330599189', version=-1) Received response(xid=36): True Sending request(xid=37): GetChildren(path='/clickhouse/tables/test/blocks/0_5823239430936010074_8336879650588380094', watcher=None) Received response(xid=37): [] Sending request(xid=38): Delete(path='/clickhouse/tables/test/blocks/0_5823239430936010074_8336879650588380094', version=-1) Received response(xid=38): True Sending request(xid=39): GetChildren(path='/clickhouse/tables/test/blocks/9_2397390479047600166_1844210758128429459', watcher=None) Received response(xid=39): [] Sending request(xid=40): Delete(path='/clickhouse/tables/test/blocks/9_2397390479047600166_1844210758128429459', version=-1) Received response(xid=40): True Sending request(xid=41): GetChildren(path='/clickhouse/tables/test/blocks/6_11800883215831824964_11063077091526655281', watcher=None) Received response(xid=41): [] Sending request(xid=42): Delete(path='/clickhouse/tables/test/blocks/6_11800883215831824964_11063077091526655281', version=-1) Received response(xid=42): True Sending request(xid=43): GetChildren(path='/clickhouse/tables/test/blocks/4_7991306156949843873_10814474702301815977', watcher=None) Received response(xid=43): [] Sending request(xid=44): Delete(path='/clickhouse/tables/test/blocks/4_7991306156949843873_10814474702301815977', version=-1) Received response(xid=44): True Sending request(xid=45): GetChildren(path='/clickhouse/tables/test/blocks/3_8943159229929697858_7063506240022641846', watcher=None) Received response(xid=45): [] Sending request(xid=46): Delete(path='/clickhouse/tables/test/blocks/3_8943159229929697858_7063506240022641846', version=-1) Received response(xid=46): True Sending request(xid=47): GetChildren(path='/clickhouse/tables/test/blocks/9_2991137436629072923_1112121919112387392', watcher=None) Received response(xid=47): [] Sending request(xid=48): Delete(path='/clickhouse/tables/test/blocks/9_2991137436629072923_1112121919112387392', version=-1) Received response(xid=48): True Sending request(xid=49): GetChildren(path='/clickhouse/tables/test/blocks/9_18111834145313106032_10223514217035097662', watcher=None) Received response(xid=49): [] Sending request(xid=50): Delete(path='/clickhouse/tables/test/blocks/9_18111834145313106032_10223514217035097662', version=-1) Received response(xid=50): True Sending request(xid=51): GetChildren(path='/clickhouse/tables/test/blocks/5_11651285249085125834_14071769460747733447', watcher=None) Received response(xid=51): [] Sending request(xid=52): Delete(path='/clickhouse/tables/test/blocks/5_11651285249085125834_14071769460747733447', version=-1) Received response(xid=52): True Sending request(xid=53): GetChildren(path='/clickhouse/tables/test/blocks/1_14567090541664558819_7702172552992592184', watcher=None) Received response(xid=53): [] Sending request(xid=54): Delete(path='/clickhouse/tables/test/blocks/1_14567090541664558819_7702172552992592184', version=-1) Received response(xid=54): True Sending request(xid=55): GetChildren(path='/clickhouse/tables/test/blocks/7_193297420037564258_6066394564520291477', watcher=None) Received response(xid=55): [] Sending request(xid=56): Delete(path='/clickhouse/tables/test/blocks/7_193297420037564258_6066394564520291477', version=-1) Received response(xid=56): True Sending request(xid=57): GetChildren(path='/clickhouse/tables/test/blocks/1_8384938112033624258_12866389235029191354', watcher=None) Received response(xid=57): [] Sending request(xid=58): Delete(path='/clickhouse/tables/test/blocks/1_8384938112033624258_12866389235029191354', version=-1) Received response(xid=58): True Sending request(xid=59): GetChildren(path='/clickhouse/tables/test/blocks/2_15871863537629735500_5393938433557777398', watcher=None) Received response(xid=59): [] Sending request(xid=60): Delete(path='/clickhouse/tables/test/blocks/2_15871863537629735500_5393938433557777398', version=-1) Received response(xid=60): True Sending request(xid=61): GetChildren(path='/clickhouse/tables/test/blocks/2_7496753474834706969_10920641722843493311', watcher=None) Received response(xid=61): [] Sending request(xid=62): Delete(path='/clickhouse/tables/test/blocks/2_7496753474834706969_10920641722843493311', version=-1) Received response(xid=62): True Sending request(xid=63): GetChildren(path='/clickhouse/tables/test/blocks/4_14956749016352276909_1845324259893722407', watcher=None) Received response(xid=63): [] Sending request(xid=64): Delete(path='/clickhouse/tables/test/blocks/4_14956749016352276909_1845324259893722407', version=-1) Received response(xid=64): True Sending request(xid=65): GetChildren(path='/clickhouse/tables/test/blocks/9_4788800582724720834_14101928124621074680', watcher=None) Received response(xid=65): [] Sending request(xid=66): Delete(path='/clickhouse/tables/test/blocks/9_4788800582724720834_14101928124621074680', version=-1) Received response(xid=66): True Sending request(xid=67): GetChildren(path='/clickhouse/tables/test/blocks/5_1736488526884124433_3887218309880210852', watcher=None) Received response(xid=67): [] Sending request(xid=68): Delete(path='/clickhouse/tables/test/blocks/5_1736488526884124433_3887218309880210852', version=-1) Received response(xid=68): True Sending request(xid=69): GetChildren(path='/clickhouse/tables/test/blocks/7_9842665926788715729_16140505362770629564', watcher=None) Received response(xid=69): [] Sending request(xid=70): Delete(path='/clickhouse/tables/test/blocks/7_9842665926788715729_16140505362770629564', version=-1) Received response(xid=70): True Sending request(xid=71): GetChildren(path='/clickhouse/tables/test/blocks/3_4110679356907143331_11035046055657096343', watcher=None) Received response(xid=71): [] Sending request(xid=72): Delete(path='/clickhouse/tables/test/blocks/3_4110679356907143331_11035046055657096343', version=-1) Received response(xid=72): True Sending request(xid=73): GetChildren(path='/clickhouse/tables/test/blocks/4_2753186338986832608_776212573121335180', watcher=None) Received response(xid=73): [] Sending request(xid=74): Delete(path='/clickhouse/tables/test/blocks/4_2753186338986832608_776212573121335180', version=-1) Received response(xid=74): True Sending request(xid=75): GetChildren(path='/clickhouse/tables/test/blocks/3_2034507573975896007_4327225584940923981', watcher=None) Received response(xid=75): [] Sending request(xid=76): Delete(path='/clickhouse/tables/test/blocks/3_2034507573975896007_4327225584940923981', version=-1) Received response(xid=76): True Sending request(xid=77): GetChildren(path='/clickhouse/tables/test/blocks/6_4626774046435431605_9754063117557495576', watcher=None) Received response(xid=77): [] Sending request(xid=78): Delete(path='/clickhouse/tables/test/blocks/6_4626774046435431605_9754063117557495576', version=-1) Received response(xid=78): True Sending request(xid=79): GetChildren(path='/clickhouse/tables/test/blocks/3_13068591570044714032_5490360431194945326', watcher=None) Received response(xid=79): [] Sending request(xid=80): Delete(path='/clickhouse/tables/test/blocks/3_13068591570044714032_5490360431194945326', version=-1) Received response(xid=80): True Sending request(xid=81): GetChildren(path='/clickhouse/tables/test/blocks/6_2088817551325874741_7861047104118916143', watcher=None) Received response(xid=81): [] Sending request(xid=82): Delete(path='/clickhouse/tables/test/blocks/6_2088817551325874741_7861047104118916143', version=-1) Received response(xid=82): True Sending request(xid=83): GetChildren(path='/clickhouse/tables/test/blocks/8_12467143234983360370_14001134191946365082', watcher=None) Received response(xid=83): [] Sending request(xid=84): Delete(path='/clickhouse/tables/test/blocks/8_12467143234983360370_14001134191946365082', version=-1) Received response(xid=84): True Sending request(xid=85): GetChildren(path='/clickhouse/tables/test/blocks/6_2855772191332027362_138014351105121520', watcher=None) Received response(xid=85): [] Sending request(xid=86): Delete(path='/clickhouse/tables/test/blocks/6_2855772191332027362_138014351105121520', version=-1) Received response(xid=86): True Sending request(xid=87): GetChildren(path='/clickhouse/tables/test/blocks/1_8011844867736689895_6905109647580634077', watcher=None) Received response(xid=87): [] Sending request(xid=88): Delete(path='/clickhouse/tables/test/blocks/1_8011844867736689895_6905109647580634077', version=-1) Received response(xid=88): True Sending request(xid=89): GetChildren(path='/clickhouse/tables/test/blocks/3_16275396704443078712_13742690842691968439', watcher=None) Received response(xid=89): [] Sending request(xid=90): Delete(path='/clickhouse/tables/test/blocks/3_16275396704443078712_13742690842691968439', version=-1) Received response(xid=90): True Sending request(xid=91): GetChildren(path='/clickhouse/tables/test/blocks/7_1015459668150584802_18117679876215610461', watcher=None) Received response(xid=91): [] Sending request(xid=92): Delete(path='/clickhouse/tables/test/blocks/7_1015459668150584802_18117679876215610461', version=-1) Received response(xid=92): True Sending request(xid=93): GetChildren(path='/clickhouse/tables/test/blocks/4_10075552373073406857_12628533709889697856', watcher=None) Received response(xid=93): [] Sending request(xid=94): Delete(path='/clickhouse/tables/test/blocks/4_10075552373073406857_12628533709889697856', version=-1) Received response(xid=94): True Sending request(xid=95): GetChildren(path='/clickhouse/tables/test/blocks/5_4397825682830605283_16079438150578917708', watcher=None) Received response(xid=95): [] Sending request(xid=96): Delete(path='/clickhouse/tables/test/blocks/5_4397825682830605283_16079438150578917708', version=-1) Received response(xid=96): True Sending request(xid=97): GetChildren(path='/clickhouse/tables/test/blocks/0_9812559225968656149_12943158647658551309', watcher=None) Received response(xid=97): [] Sending request(xid=98): Delete(path='/clickhouse/tables/test/blocks/0_9812559225968656149_12943158647658551309', version=-1) Received response(xid=98): True Sending request(xid=99): GetChildren(path='/clickhouse/tables/test/blocks/7_5315114371746958400_11285814637413696130', watcher=None) Received response(xid=99): [] Sending request(xid=100): Delete(path='/clickhouse/tables/test/blocks/7_5315114371746958400_11285814637413696130', version=-1) Received response(xid=100): True Sending request(xid=101): GetChildren(path='/clickhouse/tables/test/blocks/8_4147808760916336577_11325747988160469001', watcher=None) Received response(xid=101): [] Sending request(xid=102): Delete(path='/clickhouse/tables/test/blocks/8_4147808760916336577_11325747988160469001', version=-1) Received response(xid=102): True Sending request(xid=103): Delete(path='/clickhouse/tables/test/blocks', version=-1) Received response(xid=103): True Sending request(xid=104): GetChildren(path='/clickhouse/tables/test/part_moves_shard', watcher=None) Received response(xid=104): [] Sending request(xid=105): Delete(path='/clickhouse/tables/test/part_moves_shard', version=-1) Received response(xid=105): True Sending request(xid=106): GetChildren(path='/clickhouse/tables/test/async_blocks', watcher=None) Received response(xid=106): [] Sending request(xid=107): Delete(path='/clickhouse/tables/test/async_blocks', version=-1) Received response(xid=107): True Sending request(xid=108): GetChildren(path='/clickhouse/tables/test/log', watcher=None) Received response(xid=108): ['log-0000000029', 'log-0000000013', 'log-0000000031', 'log-0000000027', 'log-0000000028', 'log-0000000012', 'log-0000000030', 'log-0000000054', 'log-0000000055', 'log-0000000059', 'log-0000000041', 'log-0000000058', 'log-0000000040', 'log-0000000006', 'log-0000000024', 'log-0000000007', 'log-0000000025', 'log-0000000049', 'log-0000000051', 'log-0000000048', 'log-0000000050', 'log-0000000016', 'log-0000000034', 'log-0000000017', 'log-0000000039', 'log-0000000003', 'log-0000000021', 'log-0000000035', 'log-0000000038', 'log-0000000002', 'log-0000000020', 'log-0000000044', 'log-0000000045', 'log-0000000053', 'log-0000000052', 'log-0000000014', 'log-0000000036', 'log-0000000015', 'log-0000000037', 'log-0000000023', 'log-0000000019', 'log-0000000001', 'log-0000000000', 'log-0000000018', 'log-0000000022', 'log-0000000046', 'log-0000000047', 'log-0000000011', 'log-0000000009', 'log-0000000033', 'log-0000000010', 'log-0000000008', 'log-0000000032', 'log-0000000056', 'log-0000000057', 'log-0000000043', 'log-0000000042', 'log-0000000026', 'log-0000000004', 'log-0000000005'] Sending request(xid=109): GetChildren(path='/clickhouse/tables/test/log/log-0000000029', watcher=None) Received response(xid=109): [] Sending request(xid=110): Delete(path='/clickhouse/tables/test/log/log-0000000029', version=-1) http://localhost:None "GET /v1.46/containers/0b2b48351c133ca8cf267a8d2253ec0ceec17f45390656f96742901c8caa7f7b/json HTTP/1.1" 200 None Received response(xid=110): True ClickHouse node started Sending request(xid=111): GetChildren(path='/clickhouse/tables/test/log/log-0000000013', watcher=None) Received response(xid=111): [] Sending request(xid=112): Delete(path='/clickhouse/tables/test/log/log-0000000013', version=-1) Executing query CREATE TABLE distributed (id UInt32) ENGINE = Distributed('test_cluster', 'default', 'replicated') on node Received response(xid=112): True Sending request(xid=113): GetChildren(path='/clickhouse/tables/test/log/log-0000000031', watcher=None) Received response(xid=113): [] Sending request(xid=114): Delete(path='/clickhouse/tables/test/log/log-0000000031', version=-1) Received response(xid=114): True Sending request(xid=115): GetChildren(path='/clickhouse/tables/test/log/log-0000000027', watcher=None) Received response(xid=115): [] Sending request(xid=116): Delete(path='/clickhouse/tables/test/log/log-0000000027', version=-1) Received response(xid=116): True Sending request(xid=117): GetChildren(path='/clickhouse/tables/test/log/log-0000000028', watcher=None) Received response(xid=117): [] Sending request(xid=118): Delete(path='/clickhouse/tables/test/log/log-0000000028', version=-1) Received response(xid=118): True Sending request(xid=119): GetChildren(path='/clickhouse/tables/test/log/log-0000000012', watcher=None) Received response(xid=119): [] Sending request(xid=120): Delete(path='/clickhouse/tables/test/log/log-0000000012', version=-1) Received response(xid=120): True Sending request(xid=121): GetChildren(path='/clickhouse/tables/test/log/log-0000000030', watcher=None) Received response(xid=121): [] Sending request(xid=122): Delete(path='/clickhouse/tables/test/log/log-0000000030', version=-1) Received response(xid=122): True Sending request(xid=123): GetChildren(path='/clickhouse/tables/test/log/log-0000000054', watcher=None) Received response(xid=123): [] Sending request(xid=124): Delete(path='/clickhouse/tables/test/log/log-0000000054', version=-1) Received response(xid=124): True Sending request(xid=125): GetChildren(path='/clickhouse/tables/test/log/log-0000000055', watcher=None) Received response(xid=125): [] Sending request(xid=126): Delete(path='/clickhouse/tables/test/log/log-0000000055', version=-1) Received response(xid=126): True Sending request(xid=127): GetChildren(path='/clickhouse/tables/test/log/log-0000000059', watcher=None) Received response(xid=127): [] Sending request(xid=128): Delete(path='/clickhouse/tables/test/log/log-0000000059', version=-1) Received response(xid=128): True Sending request(xid=129): GetChildren(path='/clickhouse/tables/test/log/log-0000000041', watcher=None) Received response(xid=129): [] Sending request(xid=130): Delete(path='/clickhouse/tables/test/log/log-0000000041', version=-1) Received response(xid=130): True Sending request(xid=131): GetChildren(path='/clickhouse/tables/test/log/log-0000000058', watcher=None) Received response(xid=131): [] Sending request(xid=132): Delete(path='/clickhouse/tables/test/log/log-0000000058', version=-1) Received response(xid=132): True Sending request(xid=133): GetChildren(path='/clickhouse/tables/test/log/log-0000000040', watcher=None) Received response(xid=133): [] Sending request(xid=134): Delete(path='/clickhouse/tables/test/log/log-0000000040', version=-1) Received response(xid=134): True Sending request(xid=135): GetChildren(path='/clickhouse/tables/test/log/log-0000000006', watcher=None) Received response(xid=135): [] Sending request(xid=136): Delete(path='/clickhouse/tables/test/log/log-0000000006', version=-1) Received response(xid=136): True Sending request(xid=137): GetChildren(path='/clickhouse/tables/test/log/log-0000000024', watcher=None) Received response(xid=137): [] Sending request(xid=138): Delete(path='/clickhouse/tables/test/log/log-0000000024', version=-1) Received response(xid=138): True Sending request(xid=139): GetChildren(path='/clickhouse/tables/test/log/log-0000000007', watcher=None) Received response(xid=139): [] Sending request(xid=140): Delete(path='/clickhouse/tables/test/log/log-0000000007', version=-1) Received response(xid=140): True Sending request(xid=141): GetChildren(path='/clickhouse/tables/test/log/log-0000000025', watcher=None) Received response(xid=141): [] Sending request(xid=142): Delete(path='/clickhouse/tables/test/log/log-0000000025', version=-1) Received response(xid=142): True Sending request(xid=143): GetChildren(path='/clickhouse/tables/test/log/log-0000000049', watcher=None) Received response(xid=143): [] Sending request(xid=144): Delete(path='/clickhouse/tables/test/log/log-0000000049', version=-1) Received response(xid=144): True Sending request(xid=145): GetChildren(path='/clickhouse/tables/test/log/log-0000000051', watcher=None) Received response(xid=145): [] Sending request(xid=146): Delete(path='/clickhouse/tables/test/log/log-0000000051', version=-1) Received response(xid=146): True Sending request(xid=147): GetChildren(path='/clickhouse/tables/test/log/log-0000000048', watcher=None) Received response(xid=147): [] Sending request(xid=148): Delete(path='/clickhouse/tables/test/log/log-0000000048', version=-1) Received response(xid=148): True Sending request(xid=149): GetChildren(path='/clickhouse/tables/test/log/log-0000000050', watcher=None) Received response(xid=149): [] Sending request(xid=150): Delete(path='/clickhouse/tables/test/log/log-0000000050', version=-1) Received response(xid=150): True Sending request(xid=151): GetChildren(path='/clickhouse/tables/test/log/log-0000000016', watcher=None) Received response(xid=151): [] Sending request(xid=152): Delete(path='/clickhouse/tables/test/log/log-0000000016', version=-1) Received response(xid=152): True Sending request(xid=153): GetChildren(path='/clickhouse/tables/test/log/log-0000000034', watcher=None) Received response(xid=153): [] Sending request(xid=154): Delete(path='/clickhouse/tables/test/log/log-0000000034', version=-1) Received response(xid=154): True Sending request(xid=155): GetChildren(path='/clickhouse/tables/test/log/log-0000000017', watcher=None) Received response(xid=155): [] Sending request(xid=156): Delete(path='/clickhouse/tables/test/log/log-0000000017', version=-1) Received response(xid=156): True Sending request(xid=157): GetChildren(path='/clickhouse/tables/test/log/log-0000000039', watcher=None) Received response(xid=157): [] Sending request(xid=158): Delete(path='/clickhouse/tables/test/log/log-0000000039', version=-1) Received response(xid=158): True Sending request(xid=159): GetChildren(path='/clickhouse/tables/test/log/log-0000000003', watcher=None) Received response(xid=159): [] Sending request(xid=160): Delete(path='/clickhouse/tables/test/log/log-0000000003', version=-1) Received response(xid=160): True Sending request(xid=161): GetChildren(path='/clickhouse/tables/test/log/log-0000000021', watcher=None) Received response(xid=161): [] Sending request(xid=162): Delete(path='/clickhouse/tables/test/log/log-0000000021', version=-1) Received response(xid=162): True Sending request(xid=163): GetChildren(path='/clickhouse/tables/test/log/log-0000000035', watcher=None) Received response(xid=163): [] Sending request(xid=164): Delete(path='/clickhouse/tables/test/log/log-0000000035', version=-1) Received response(xid=164): True Sending request(xid=165): GetChildren(path='/clickhouse/tables/test/log/log-0000000038', watcher=None) Received response(xid=165): [] Sending request(xid=166): Delete(path='/clickhouse/tables/test/log/log-0000000038', version=-1) Received response(xid=166): True Sending request(xid=167): GetChildren(path='/clickhouse/tables/test/log/log-0000000002', watcher=None) Received response(xid=167): [] Sending request(xid=168): Delete(path='/clickhouse/tables/test/log/log-0000000002', version=-1) Executing query CREATE TABLE distributed2 (id UInt32) ENGINE = Distributed('test_cluster2', 'default', 'replicated') on node Received response(xid=168): True Sending request(xid=169): GetChildren(path='/clickhouse/tables/test/log/log-0000000020', watcher=None) Received response(xid=169): [] Sending request(xid=170): Delete(path='/clickhouse/tables/test/log/log-0000000020', version=-1) Received response(xid=170): True Sending request(xid=171): GetChildren(path='/clickhouse/tables/test/log/log-0000000044', watcher=None) Received response(xid=171): [] Sending request(xid=172): Delete(path='/clickhouse/tables/test/log/log-0000000044', version=-1) Received response(xid=172): True Sending request(xid=173): GetChildren(path='/clickhouse/tables/test/log/log-0000000045', watcher=None) Received response(xid=173): [] Sending request(xid=174): Delete(path='/clickhouse/tables/test/log/log-0000000045', version=-1) Received response(xid=174): True Sending request(xid=175): GetChildren(path='/clickhouse/tables/test/log/log-0000000053', watcher=None) Received response(xid=175): [] Sending request(xid=176): Delete(path='/clickhouse/tables/test/log/log-0000000053', version=-1) Received response(xid=176): True Sending request(xid=177): GetChildren(path='/clickhouse/tables/test/log/log-0000000052', watcher=None) Received response(xid=177): [] Sending request(xid=178): Delete(path='/clickhouse/tables/test/log/log-0000000052', version=-1) Received response(xid=178): True Sending request(xid=179): GetChildren(path='/clickhouse/tables/test/log/log-0000000014', watcher=None) Received response(xid=179): [] Sending request(xid=180): Delete(path='/clickhouse/tables/test/log/log-0000000014', version=-1) Received response(xid=180): True Sending request(xid=181): GetChildren(path='/clickhouse/tables/test/log/log-0000000036', watcher=None) Received response(xid=181): [] Sending request(xid=182): Delete(path='/clickhouse/tables/test/log/log-0000000036', version=-1) Received response(xid=182): True Sending request(xid=183): GetChildren(path='/clickhouse/tables/test/log/log-0000000015', watcher=None) Received response(xid=183): [] Sending request(xid=184): Delete(path='/clickhouse/tables/test/log/log-0000000015', version=-1) Received response(xid=184): True Sending request(xid=185): GetChildren(path='/clickhouse/tables/test/log/log-0000000037', watcher=None) Received response(xid=185): [] Sending request(xid=186): Delete(path='/clickhouse/tables/test/log/log-0000000037', version=-1) Received response(xid=186): True Sending request(xid=187): GetChildren(path='/clickhouse/tables/test/log/log-0000000023', watcher=None) Received response(xid=187): [] Sending request(xid=188): Delete(path='/clickhouse/tables/test/log/log-0000000023', version=-1) Received response(xid=188): True Sending request(xid=189): GetChildren(path='/clickhouse/tables/test/log/log-0000000019', watcher=None) Received response(xid=189): [] Sending request(xid=190): Delete(path='/clickhouse/tables/test/log/log-0000000019', version=-1) Received response(xid=190): True Sending request(xid=191): GetChildren(path='/clickhouse/tables/test/log/log-0000000001', watcher=None) Received response(xid=191): [] Sending request(xid=192): Delete(path='/clickhouse/tables/test/log/log-0000000001', version=-1) Received response(xid=192): True Sending request(xid=193): GetChildren(path='/clickhouse/tables/test/log/log-0000000000', watcher=None) Received response(xid=193): [] Sending request(xid=194): Delete(path='/clickhouse/tables/test/log/log-0000000000', version=-1) Received response(xid=194): True Sending request(xid=195): GetChildren(path='/clickhouse/tables/test/log/log-0000000018', watcher=None) Received response(xid=195): [] Sending request(xid=196): Delete(path='/clickhouse/tables/test/log/log-0000000018', version=-1) Received response(xid=196): True Sending request(xid=197): GetChildren(path='/clickhouse/tables/test/log/log-0000000022', watcher=None) Received response(xid=197): [] Sending request(xid=198): Delete(path='/clickhouse/tables/test/log/log-0000000022', version=-1) Received response(xid=198): True Sending request(xid=199): GetChildren(path='/clickhouse/tables/test/log/log-0000000046', watcher=None) Received response(xid=199): [] Sending request(xid=200): Delete(path='/clickhouse/tables/test/log/log-0000000046', version=-1) Received response(xid=200): True Sending request(xid=201): GetChildren(path='/clickhouse/tables/test/log/log-0000000047', watcher=None) Received response(xid=201): [] Sending request(xid=202): Delete(path='/clickhouse/tables/test/log/log-0000000047', version=-1) Received response(xid=202): True Sending request(xid=203): GetChildren(path='/clickhouse/tables/test/log/log-0000000011', watcher=None) Received response(xid=203): [] Sending request(xid=204): Delete(path='/clickhouse/tables/test/log/log-0000000011', version=-1) Received response(xid=204): True Sending request(xid=205): GetChildren(path='/clickhouse/tables/test/log/log-0000000009', watcher=None) Received response(xid=205): [] Sending request(xid=206): Delete(path='/clickhouse/tables/test/log/log-0000000009', version=-1) Received response(xid=206): True Sending request(xid=207): GetChildren(path='/clickhouse/tables/test/log/log-0000000033', watcher=None) Received response(xid=207): [] Sending request(xid=208): Delete(path='/clickhouse/tables/test/log/log-0000000033', version=-1) Received response(xid=208): True Sending request(xid=209): GetChildren(path='/clickhouse/tables/test/log/log-0000000010', watcher=None) Received response(xid=209): [] Sending request(xid=210): Delete(path='/clickhouse/tables/test/log/log-0000000010', version=-1) Received response(xid=210): True Sending request(xid=211): GetChildren(path='/clickhouse/tables/test/log/log-0000000008', watcher=None) Received response(xid=211): [] Sending request(xid=212): Delete(path='/clickhouse/tables/test/log/log-0000000008', version=-1) Received response(xid=212): True Sending request(xid=213): GetChildren(path='/clickhouse/tables/test/log/log-0000000032', watcher=None) Received response(xid=213): [] Sending request(xid=214): Delete(path='/clickhouse/tables/test/log/log-0000000032', version=-1) Received response(xid=214): True Sending request(xid=215): GetChildren(path='/clickhouse/tables/test/log/log-0000000056', watcher=None) Received response(xid=215): [] Sending request(xid=216): Delete(path='/clickhouse/tables/test/log/log-0000000056', version=-1) Received response(xid=216): True Sending request(xid=217): GetChildren(path='/clickhouse/tables/test/log/log-0000000057', watcher=None) Received response(xid=217): [] Sending request(xid=218): Delete(path='/clickhouse/tables/test/log/log-0000000057', version=-1) Received response(xid=218): True Sending request(xid=219): GetChildren(path='/clickhouse/tables/test/log/log-0000000043', watcher=None) Received response(xid=219): [] Sending request(xid=220): Delete(path='/clickhouse/tables/test/log/log-0000000043', version=-1) Received response(xid=220): True Sending request(xid=221): GetChildren(path='/clickhouse/tables/test/log/log-0000000042', watcher=None) Received response(xid=221): [] Sending request(xid=222): Delete(path='/clickhouse/tables/test/log/log-0000000042', version=-1) Received response(xid=222): True Sending request(xid=223): GetChildren(path='/clickhouse/tables/test/log/log-0000000026', watcher=None) Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Received response(xid=223): [] Sending request(xid=224): Delete(path='/clickhouse/tables/test/log/log-0000000026', version=-1) Received response(xid=224): True Sending request(xid=225): GetChildren(path='/clickhouse/tables/test/log/log-0000000004', watcher=None) Received response(xid=225): [] Sending request(xid=226): Delete(path='/clickhouse/tables/test/log/log-0000000004', version=-1) Received response(xid=226): True Sending request(xid=227): GetChildren(path='/clickhouse/tables/test/log/log-0000000005', watcher=None) Received response(xid=227): [] Sending request(xid=228): Delete(path='/clickhouse/tables/test/log/log-0000000005', version=-1) Received response(xid=228): True Sending request(xid=229): Delete(path='/clickhouse/tables/test/log', version=-1) Received response(xid=229): True Sending request(xid=230): GetChildren(path='/clickhouse/tables/test/replicas', watcher=None) Received response(xid=230): ['replica2', 'replica1', 'replica3'] Sending request(xid=231): GetChildren(path='/clickhouse/tables/test/replicas/replica2', watcher=None) Received response(xid=231): ['flags', 'is_lost', 'metadata', 'max_processed_insert_time', 'log_pointer', 'parts', 'queue', 'host', 'metadata_version', 'columns', 'mutation_pointer', 'min_unprocessed_insert_time', 'creator_info', 'is_active'] Sending request(xid=232): GetChildren(path='/clickhouse/tables/test/replicas/replica2/flags', watcher=None) Received response(xid=232): [] Sending request(xid=233): Delete(path='/clickhouse/tables/test/replicas/replica2/flags', version=-1) Received response(xid=233): True Sending request(xid=234): GetChildren(path='/clickhouse/tables/test/replicas/replica2/is_lost', watcher=None) Received response(xid=234): [] Sending request(xid=235): Delete(path='/clickhouse/tables/test/replicas/replica2/is_lost', version=-1) Received response(xid=235): True Sending request(xid=236): GetChildren(path='/clickhouse/tables/test/replicas/replica2/metadata', watcher=None) Received response(xid=236): [] Sending request(xid=237): Delete(path='/clickhouse/tables/test/replicas/replica2/metadata', version=-1) Received response(xid=237): True Sending request(xid=238): GetChildren(path='/clickhouse/tables/test/replicas/replica2/max_processed_insert_time', watcher=None) Received response(xid=238): [] Sending request(xid=239): Delete(path='/clickhouse/tables/test/replicas/replica2/max_processed_insert_time', version=-1) Received response(xid=239): True Sending request(xid=240): GetChildren(path='/clickhouse/tables/test/replicas/replica2/log_pointer', watcher=None) Received response(xid=240): [] Sending request(xid=241): Delete(path='/clickhouse/tables/test/replicas/replica2/log_pointer', version=-1) Received response(xid=241): True Sending request(xid=242): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts', watcher=None) Received response(xid=242): ['3_2_2_0', '4_4_4_0', '5_0_4_1', '3_4_4_0', '7_0_0_0', '3_3_3_0', '9_4_4_0', '6_1_1_0', '5_1_1_0', '1_4_4_0', '7_3_3_0', '1_3_3_0', '9_0_0_0', '4_3_3_0', '8_4_4_0', '9_1_1_0', '7_1_1_0', '4_0_4_1', '2_3_3_0', '4_2_2_0', '6_0_4_1', '3_0_4_1', '1_2_2_0', '9_2_2_0', '0_0_4_1', '6_3_3_0', '6_2_2_0', '0_4_4_0', '6_0_0_0', '4_0_0_0', '5_0_0_0', '0_0_0_0', '1_1_1_0', '3_1_1_0', '8_0_4_1', '5_3_3_0', '4_1_1_0', '2_2_2_0', '0_1_1_0', '9_3_3_0', '5_4_4_0', '8_1_1_0', '2_0_4_1', '2_0_0_0', '8_2_2_0', '8_3_3_0', '1_0_0_0', '2_1_1_0', '0_3_3_0', '0_2_2_0', '6_4_4_0', '2_4_4_0', '7_0_4_1', '7_4_4_0', '1_0_4_1', '7_2_2_0', '9_0_4_1', '8_0_0_0', '5_2_2_0', '3_0_0_0'] Sending request(xid=243): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/3_2_2_0', watcher=None) Received response(xid=243): [] Sending request(xid=244): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/3_2_2_0', version=-1) Received response(xid=244): True Sending request(xid=245): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/4_4_4_0', watcher=None) Received response(xid=245): [] Sending request(xid=246): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/4_4_4_0', version=-1) Received response(xid=246): True Sending request(xid=247): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/5_0_4_1', watcher=None) Received response(xid=247): [] Sending request(xid=248): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/5_0_4_1', version=-1) Received response(xid=248): True Sending request(xid=249): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/3_4_4_0', watcher=None) Received response(xid=249): [] Sending request(xid=250): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/3_4_4_0', version=-1) Received response(xid=250): True Sending request(xid=251): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/7_0_0_0', watcher=None) Received response(xid=251): [] Sending request(xid=252): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/7_0_0_0', version=-1) Received response(xid=252): True Sending request(xid=253): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/3_3_3_0', watcher=None) Received response(xid=253): [] Sending request(xid=254): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/3_3_3_0', version=-1) Received response(xid=254): True Sending request(xid=255): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/9_4_4_0', watcher=None) Received response(xid=255): [] Sending request(xid=256): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/9_4_4_0', version=-1) Received response(xid=256): True Sending request(xid=257): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/6_1_1_0', watcher=None) Received response(xid=257): [] Sending request(xid=258): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/6_1_1_0', version=-1) Received response(xid=258): True Sending request(xid=259): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/5_1_1_0', watcher=None) Received response(xid=259): [] Sending request(xid=260): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/5_1_1_0', version=-1) Received response(xid=260): True Sending request(xid=261): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/1_4_4_0', watcher=None) Received response(xid=261): [] Sending request(xid=262): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/1_4_4_0', version=-1) Received response(xid=262): True Sending request(xid=263): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/7_3_3_0', watcher=None) Received response(xid=263): [] Sending request(xid=264): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/7_3_3_0', version=-1) Received response(xid=264): True Sending request(xid=265): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/1_3_3_0', watcher=None) Received response(xid=265): [] Sending request(xid=266): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/1_3_3_0', version=-1) Received response(xid=266): True Sending request(xid=267): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/9_0_0_0', watcher=None) Received response(xid=267): [] Sending request(xid=268): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/9_0_0_0', version=-1) Received response(xid=268): True Sending request(xid=269): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/4_3_3_0', watcher=None) Received response(xid=269): [] Sending request(xid=270): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/4_3_3_0', version=-1) Received response(xid=270): True Sending request(xid=271): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/8_4_4_0', watcher=None) Received response(xid=271): [] Sending request(xid=272): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/8_4_4_0', version=-1) Received response(xid=272): True Sending request(xid=273): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/9_1_1_0', watcher=None) Received response(xid=273): [] Sending request(xid=274): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/9_1_1_0', version=-1) Received response(xid=274): True Sending request(xid=275): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/7_1_1_0', watcher=None) Received response(xid=275): [] Sending request(xid=276): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/7_1_1_0', version=-1) Received response(xid=276): True Sending request(xid=277): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/4_0_4_1', watcher=None) Received response(xid=277): [] Sending request(xid=278): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/4_0_4_1', version=-1) Received response(xid=278): True Sending request(xid=279): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/2_3_3_0', watcher=None) Received response(xid=279): [] Sending request(xid=280): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/2_3_3_0', version=-1) Received response(xid=280): True Sending request(xid=281): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/4_2_2_0', watcher=None) Received response(xid=281): [] Sending request(xid=282): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/4_2_2_0', version=-1) Received response(xid=282): True Sending request(xid=283): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/6_0_4_1', watcher=None) Received response(xid=283): [] Sending request(xid=284): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/6_0_4_1', version=-1) Received response(xid=284): True Sending request(xid=285): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/3_0_4_1', watcher=None) Received response(xid=285): [] Sending request(xid=286): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/3_0_4_1', version=-1) Received response(xid=286): True Sending request(xid=287): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/1_2_2_0', watcher=None) Received response(xid=287): [] Sending request(xid=288): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/1_2_2_0', version=-1) Received response(xid=288): True Sending request(xid=289): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/9_2_2_0', watcher=None) Received response(xid=289): [] Sending request(xid=290): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/9_2_2_0', version=-1) Received response(xid=290): True Sending request(xid=291): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/0_0_4_1', watcher=None) Received response(xid=291): [] Sending request(xid=292): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/0_0_4_1', version=-1) run container_id:roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Received response(xid=292): True Sending request(xid=293): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/6_3_3_0', watcher=None) Received response(xid=293): [] Sending request(xid=294): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/6_3_3_0', version=-1) Received response(xid=294): True Sending request(xid=295): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/6_2_2_0', watcher=None) Received response(xid=295): [] Sending request(xid=296): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/6_2_2_0', version=-1) Received response(xid=296): True Sending request(xid=297): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/0_4_4_0', watcher=None) Received response(xid=297): [] Sending request(xid=298): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/0_4_4_0', version=-1) Received response(xid=298): True Sending request(xid=299): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/6_0_0_0', watcher=None) Received response(xid=299): [] Sending request(xid=300): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/6_0_0_0', version=-1) Received response(xid=300): True Sending request(xid=301): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/4_0_0_0', watcher=None) Received response(xid=301): [] Sending request(xid=302): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/4_0_0_0', version=-1) Received response(xid=302): True Sending request(xid=303): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/5_0_0_0', watcher=None) Received response(xid=303): [] Sending request(xid=304): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/5_0_0_0', version=-1) Received response(xid=304): True Sending request(xid=305): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/0_0_0_0', watcher=None) Received response(xid=305): [] Sending request(xid=306): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/0_0_0_0', version=-1) Received response(xid=306): True Sending request(xid=307): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/1_1_1_0', watcher=None) Received response(xid=307): [] Sending request(xid=308): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/1_1_1_0', version=-1) Received response(xid=308): True Sending request(xid=309): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/3_1_1_0', watcher=None) Received response(xid=309): [] Sending request(xid=310): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/3_1_1_0', version=-1) Received response(xid=310): True Sending request(xid=311): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/8_0_4_1', watcher=None) Received response(xid=311): [] Sending request(xid=312): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/8_0_4_1', version=-1) Received response(xid=312): True Sending request(xid=313): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/5_3_3_0', watcher=None) Received response(xid=313): [] Sending request(xid=314): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/5_3_3_0', version=-1) Received response(xid=314): True Sending request(xid=315): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/4_1_1_0', watcher=None) Received response(xid=315): [] Sending request(xid=316): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/4_1_1_0', version=-1) Received response(xid=316): True Sending request(xid=317): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/2_2_2_0', watcher=None) Received response(xid=317): [] Sending request(xid=318): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/2_2_2_0', version=-1) Received response(xid=318): True Sending request(xid=319): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/0_1_1_0', watcher=None) Received response(xid=319): [] Sending request(xid=320): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/0_1_1_0', version=-1) Received response(xid=320): True Sending request(xid=321): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/9_3_3_0', watcher=None) Received response(xid=321): [] Sending request(xid=322): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/9_3_3_0', version=-1) Received response(xid=322): True Sending request(xid=323): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/5_4_4_0', watcher=None) Received response(xid=323): [] Sending request(xid=324): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/5_4_4_0', version=-1) Received response(xid=324): True Sending request(xid=325): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/8_1_1_0', watcher=None) Received response(xid=325): [] Sending request(xid=326): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/8_1_1_0', version=-1) Received response(xid=326): True Sending request(xid=327): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/2_0_4_1', watcher=None) Received response(xid=327): [] Sending request(xid=328): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/2_0_4_1', version=-1) Received response(xid=328): True Sending request(xid=329): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/2_0_0_0', watcher=None) Received response(xid=329): [] Sending request(xid=330): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/2_0_0_0', version=-1) Received response(xid=330): True Sending request(xid=331): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/8_2_2_0', watcher=None) Received response(xid=331): [] Sending request(xid=332): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/8_2_2_0', version=-1) Received response(xid=332): True Sending request(xid=333): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/8_3_3_0', watcher=None) Received response(xid=333): [] Sending request(xid=334): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/8_3_3_0', version=-1) Received response(xid=334): True Stdout:812 Clickhouse process running. Sending request(xid=335): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/1_0_0_0', watcher=None) run container_id:roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Received response(xid=335): [] Sending request(xid=336): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/1_0_0_0', version=-1) Received response(xid=336): True Sending request(xid=337): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/2_1_1_0', watcher=None) Received response(xid=337): [] Sending request(xid=338): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/2_1_1_0', version=-1) Received response(xid=338): True Sending request(xid=339): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/0_3_3_0', watcher=None) Received response(xid=339): [] Sending request(xid=340): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/0_3_3_0', version=-1) Received response(xid=340): True Sending request(xid=341): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/0_2_2_0', watcher=None) Received response(xid=341): [] Sending request(xid=342): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/0_2_2_0', version=-1) Received response(xid=342): True Sending request(xid=343): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/6_4_4_0', watcher=None) Received response(xid=343): [] Sending request(xid=344): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/6_4_4_0', version=-1) Received response(xid=344): True Sending request(xid=345): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/2_4_4_0', watcher=None) Received response(xid=345): [] Sending request(xid=346): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/2_4_4_0', version=-1) Received response(xid=346): True Sending request(xid=347): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/7_0_4_1', watcher=None) Received response(xid=347): [] Sending request(xid=348): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/7_0_4_1', version=-1) Received response(xid=348): True Sending request(xid=349): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/7_4_4_0', watcher=None) Received response(xid=349): [] Sending request(xid=350): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/7_4_4_0', version=-1) Received response(xid=350): True Sending request(xid=351): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/1_0_4_1', watcher=None) Received response(xid=351): [] Sending request(xid=352): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/1_0_4_1', version=-1) Received response(xid=352): True Sending request(xid=353): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/7_2_2_0', watcher=None) Received response(xid=353): [] Sending request(xid=354): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/7_2_2_0', version=-1) Received response(xid=354): True Sending request(xid=355): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/9_0_4_1', watcher=None) Received response(xid=355): [] Sending request(xid=356): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/9_0_4_1', version=-1) Received response(xid=356): True Sending request(xid=357): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/8_0_0_0', watcher=None) Received response(xid=357): [] Sending request(xid=358): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/8_0_0_0', version=-1) Received response(xid=358): True Sending request(xid=359): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/5_2_2_0', watcher=None) Received response(xid=359): [] Sending request(xid=360): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/5_2_2_0', version=-1) Received response(xid=360): True Sending request(xid=361): GetChildren(path='/clickhouse/tables/test/replicas/replica2/parts/3_0_0_0', watcher=None) Received response(xid=361): [] Sending request(xid=362): Delete(path='/clickhouse/tables/test/replicas/replica2/parts/3_0_0_0', version=-1) Received response(xid=362): True Sending request(xid=363): Delete(path='/clickhouse/tables/test/replicas/replica2/parts', version=-1) Received response(xid=363): True Sending request(xid=364): GetChildren(path='/clickhouse/tables/test/replicas/replica2/queue', watcher=None) Received response(xid=364): [] Sending request(xid=365): Delete(path='/clickhouse/tables/test/replicas/replica2/queue', version=-1) Received response(xid=365): True Sending request(xid=366): GetChildren(path='/clickhouse/tables/test/replicas/replica2/host', watcher=None) Received response(xid=366): [] Sending request(xid=367): Delete(path='/clickhouse/tables/test/replicas/replica2/host', version=-1) Received response(xid=367): True Sending request(xid=368): GetChildren(path='/clickhouse/tables/test/replicas/replica2/metadata_version', watcher=None) Received response(xid=368): [] Sending request(xid=369): Delete(path='/clickhouse/tables/test/replicas/replica2/metadata_version', version=-1) Received response(xid=369): True Sending request(xid=370): GetChildren(path='/clickhouse/tables/test/replicas/replica2/columns', watcher=None) Received response(xid=370): [] Sending request(xid=371): Delete(path='/clickhouse/tables/test/replicas/replica2/columns', version=-1) Stdout:812 Executing query select 20 on switching_node Received response(xid=371): True Sending request(xid=372): GetChildren(path='/clickhouse/tables/test/replicas/replica2/mutation_pointer', watcher=None) Received response(xid=372): [] Sending request(xid=373): Delete(path='/clickhouse/tables/test/replicas/replica2/mutation_pointer', version=-1) Received response(xid=373): True Sending request(xid=374): GetChildren(path='/clickhouse/tables/test/replicas/replica2/min_unprocessed_insert_time', watcher=None) Received response(xid=374): [] Sending request(xid=375): Delete(path='/clickhouse/tables/test/replicas/replica2/min_unprocessed_insert_time', version=-1) Received response(xid=375): True Sending request(xid=376): GetChildren(path='/clickhouse/tables/test/replicas/replica2/creator_info', watcher=None) Received response(xid=376): [] Sending request(xid=377): Delete(path='/clickhouse/tables/test/replicas/replica2/creator_info', version=-1) Received response(xid=377): True Sending request(xid=378): GetChildren(path='/clickhouse/tables/test/replicas/replica2/is_active', watcher=None) Received response(xid=378): [] Sending request(xid=379): Delete(path='/clickhouse/tables/test/replicas/replica2/is_active', version=-1) Received response(xid=379): True Sending request(xid=380): Delete(path='/clickhouse/tables/test/replicas/replica2', version=-1) Received response(xid=380): True Sending request(xid=381): GetChildren(path='/clickhouse/tables/test/replicas/replica1', watcher=None) Received response(xid=381): ['flags', 'is_lost', 'metadata', 'columns', 'queue', 'parts', 'log_pointer', 'host', 'min_unprocessed_insert_time', 'max_processed_insert_time', 'metadata_version', 'mutation_pointer', 'creator_info', 'is_active'] Sending request(xid=382): GetChildren(path='/clickhouse/tables/test/replicas/replica1/flags', watcher=None) Received response(xid=382): [] Sending request(xid=383): Delete(path='/clickhouse/tables/test/replicas/replica1/flags', version=-1) Received response(xid=383): True Sending request(xid=384): GetChildren(path='/clickhouse/tables/test/replicas/replica1/is_lost', watcher=None) Received response(xid=384): [] Sending request(xid=385): Delete(path='/clickhouse/tables/test/replicas/replica1/is_lost', version=-1) Received response(xid=385): True Sending request(xid=386): GetChildren(path='/clickhouse/tables/test/replicas/replica1/metadata', watcher=None) Received response(xid=386): [] Sending request(xid=387): Delete(path='/clickhouse/tables/test/replicas/replica1/metadata', version=-1) Received response(xid=387): True Sending request(xid=388): GetChildren(path='/clickhouse/tables/test/replicas/replica1/columns', watcher=None) Received response(xid=388): [] Sending request(xid=389): Delete(path='/clickhouse/tables/test/replicas/replica1/columns', version=-1) Received response(xid=389): True Sending request(xid=390): GetChildren(path='/clickhouse/tables/test/replicas/replica1/queue', watcher=None) Received response(xid=390): [] Sending request(xid=391): Delete(path='/clickhouse/tables/test/replicas/replica1/queue', version=-1) Received response(xid=391): True Sending request(xid=392): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts', watcher=None) Received response(xid=392): ['3_0_0_0', '3_2_2_0', '5_0_4_1', '4_4_4_0', '3_4_4_0', '7_0_0_0', '9_4_4_0', '3_3_3_0', '6_1_1_0', '5_1_1_0', '7_3_3_0', '1_4_4_0', '1_3_3_0', '4_3_3_0', '9_0_0_0', '8_4_4_0', '9_1_1_0', '7_1_1_0', '4_0_4_1', '4_2_2_0', '2_3_3_0', '6_0_4_1', '9_2_2_0', '3_0_4_1', '1_2_2_0', '6_3_3_0', '0_0_4_1', '6_2_2_0', '0_4_4_0', '4_0_0_0', '6_0_0_0', '0_0_0_0', '5_0_0_0', '1_1_1_0', '3_1_1_0', '8_0_4_1', '5_3_3_0', '4_1_1_0', '2_2_2_0', '0_1_1_0', '9_3_3_0', '5_4_4_0', '8_1_1_0', '2_0_4_1', '2_0_0_0', '8_2_2_0', '8_3_3_0', '1_0_0_0', '0_2_2_0', '2_1_1_0', '2_4_4_0', '0_3_3_0', '7_4_4_0', '6_4_4_0', '7_0_4_1', '1_0_4_1', '7_2_2_0', '9_0_4_1', '8_0_0_0', '5_2_2_0'] Sending request(xid=393): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/3_0_0_0', watcher=None) Received response(xid=393): [] Sending request(xid=394): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/3_0_0_0', version=-1) Received response(xid=394): True Sending request(xid=395): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/3_2_2_0', watcher=None) Received response(xid=395): [] Sending request(xid=396): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/3_2_2_0', version=-1) Received response(xid=396): True Sending request(xid=397): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/5_0_4_1', watcher=None) Received response(xid=397): [] Sending request(xid=398): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/5_0_4_1', version=-1) Received response(xid=398): True Sending request(xid=399): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/4_4_4_0', watcher=None) Received response(xid=399): [] Sending request(xid=400): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/4_4_4_0', version=-1) Received response(xid=400): True Sending request(xid=401): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/3_4_4_0', watcher=None) Received response(xid=401): [] Sending request(xid=402): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/3_4_4_0', version=-1) Received response(xid=402): True Sending request(xid=403): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/7_0_0_0', watcher=None) Received response(xid=403): [] Sending request(xid=404): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/7_0_0_0', version=-1) Received response(xid=404): True Sending request(xid=405): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/9_4_4_0', watcher=None) Received response(xid=405): [] Sending request(xid=406): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/9_4_4_0', version=-1) Received response(xid=406): True Sending request(xid=407): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/3_3_3_0', watcher=None) Received response(xid=407): [] Sending request(xid=408): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/3_3_3_0', version=-1) Received response(xid=408): True Sending request(xid=409): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/6_1_1_0', watcher=None) Received response(xid=409): [] Sending request(xid=410): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/6_1_1_0', version=-1) Received response(xid=410): True Sending request(xid=411): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/5_1_1_0', watcher=None) Received response(xid=411): [] Sending request(xid=412): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/5_1_1_0', version=-1) Received response(xid=412): True Sending request(xid=413): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/7_3_3_0', watcher=None) Received response(xid=413): [] Sending request(xid=414): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/7_3_3_0', version=-1) Received response(xid=414): True Sending request(xid=415): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/1_4_4_0', watcher=None) Received response(xid=415): [] Sending request(xid=416): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/1_4_4_0', version=-1) Received response(xid=416): True Sending request(xid=417): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/1_3_3_0', watcher=None) Received response(xid=417): [] Sending request(xid=418): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/1_3_3_0', version=-1) Received response(xid=418): True Sending request(xid=419): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/4_3_3_0', watcher=None) Received response(xid=419): [] Sending request(xid=420): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/4_3_3_0', version=-1) Received response(xid=420): True Sending request(xid=421): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/9_0_0_0', watcher=None) Received response(xid=421): [] Sending request(xid=422): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/9_0_0_0', version=-1) Received response(xid=422): True Sending request(xid=423): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/8_4_4_0', watcher=None) Received response(xid=423): [] Sending request(xid=424): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/8_4_4_0', version=-1) Received response(xid=424): True Sending request(xid=425): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/9_1_1_0', watcher=None) Received response(xid=425): [] Sending request(xid=426): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/9_1_1_0', version=-1) Received response(xid=426): True Sending request(xid=427): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/7_1_1_0', watcher=None) Received response(xid=427): [] Sending request(xid=428): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/7_1_1_0', version=-1) Received response(xid=428): True Sending request(xid=429): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/4_0_4_1', watcher=None) Received response(xid=429): [] Sending request(xid=430): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/4_0_4_1', version=-1) Received response(xid=430): True Sending request(xid=431): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/4_2_2_0', watcher=None) Received response(xid=431): [] Sending request(xid=432): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/4_2_2_0', version=-1) Received response(xid=432): True Sending request(xid=433): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/2_3_3_0', watcher=None) Received response(xid=433): [] Sending request(xid=434): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/2_3_3_0', version=-1) Received response(xid=434): True Sending request(xid=435): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/6_0_4_1', watcher=None) Executing query INSERT INTO test_log_table VALUES (0, 'a') on switching_node Received response(xid=435): [] Sending request(xid=436): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/6_0_4_1', version=-1) Received response(xid=436): True Sending request(xid=437): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/9_2_2_0', watcher=None) Received response(xid=437): [] Sending request(xid=438): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/9_2_2_0', version=-1) Received response(xid=438): True Sending request(xid=439): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/3_0_4_1', watcher=None) Received response(xid=439): [] Sending request(xid=440): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/3_0_4_1', version=-1) Received response(xid=440): True Sending request(xid=441): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/1_2_2_0', watcher=None) Received response(xid=441): [] Sending request(xid=442): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/1_2_2_0', version=-1) Received response(xid=442): True Sending request(xid=443): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/6_3_3_0', watcher=None) Received response(xid=443): [] Sending request(xid=444): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/6_3_3_0', version=-1) Received response(xid=444): True Sending request(xid=445): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/0_0_4_1', watcher=None) Received response(xid=445): [] Sending request(xid=446): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/0_0_4_1', version=-1) Received response(xid=446): True Sending request(xid=447): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/6_2_2_0', watcher=None) Received response(xid=447): [] Sending request(xid=448): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/6_2_2_0', version=-1) Received response(xid=448): True Sending request(xid=449): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/0_4_4_0', watcher=None) Received response(xid=449): [] Sending request(xid=450): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/0_4_4_0', version=-1) Received response(xid=450): True Sending request(xid=451): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/4_0_0_0', watcher=None) Received response(xid=451): [] Sending request(xid=452): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/4_0_0_0', version=-1) Received response(xid=452): True Sending request(xid=453): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/6_0_0_0', watcher=None) Received response(xid=453): [] Sending request(xid=454): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/6_0_0_0', version=-1) Received response(xid=454): True Sending request(xid=455): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/0_0_0_0', watcher=None) Received response(xid=455): [] Sending request(xid=456): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/0_0_0_0', version=-1) Received response(xid=456): True Sending request(xid=457): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/5_0_0_0', watcher=None) Received response(xid=457): [] Sending request(xid=458): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/5_0_0_0', version=-1) Received response(xid=458): True Sending request(xid=459): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/1_1_1_0', watcher=None) Received response(xid=459): [] Sending request(xid=460): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/1_1_1_0', version=-1) Received response(xid=460): True Sending request(xid=461): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/3_1_1_0', watcher=None) Received response(xid=461): [] Sending request(xid=462): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/3_1_1_0', version=-1) Received response(xid=462): True Sending request(xid=463): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/8_0_4_1', watcher=None) Received response(xid=463): [] Sending request(xid=464): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/8_0_4_1', version=-1) Received response(xid=464): True Sending request(xid=465): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/5_3_3_0', watcher=None) Received response(xid=465): [] Sending request(xid=466): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/5_3_3_0', version=-1) Received response(xid=466): True Sending request(xid=467): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/4_1_1_0', watcher=None) Received response(xid=467): [] Sending request(xid=468): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/4_1_1_0', version=-1) Received response(xid=468): True Sending request(xid=469): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/2_2_2_0', watcher=None) Received response(xid=469): [] Sending request(xid=470): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/2_2_2_0', version=-1) Received response(xid=470): True Sending request(xid=471): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/0_1_1_0', watcher=None) Received response(xid=471): [] Sending request(xid=472): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/0_1_1_0', version=-1) Received response(xid=472): True Sending request(xid=473): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/9_3_3_0', watcher=None) Received response(xid=473): [] Sending request(xid=474): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/9_3_3_0', version=-1) Received response(xid=474): True Sending request(xid=475): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/5_4_4_0', watcher=None) Received response(xid=475): [] Sending request(xid=476): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/5_4_4_0', version=-1) Received response(xid=476): True Sending request(xid=477): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/8_1_1_0', watcher=None) Received response(xid=477): [] Sending request(xid=478): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/8_1_1_0', version=-1) Received response(xid=478): True Sending request(xid=479): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/2_0_4_1', watcher=None) Received response(xid=479): [] Sending request(xid=480): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/2_0_4_1', version=-1) Received response(xid=480): True Sending request(xid=481): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/2_0_0_0', watcher=None) Received response(xid=481): [] Sending request(xid=482): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/2_0_0_0', version=-1) Received response(xid=482): True Sending request(xid=483): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/8_2_2_0', watcher=None) Received response(xid=483): [] Sending request(xid=484): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/8_2_2_0', version=-1) Received response(xid=484): True Sending request(xid=485): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/8_3_3_0', watcher=None) Received response(xid=485): [] Sending request(xid=486): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/8_3_3_0', version=-1) Received response(xid=486): True Sending request(xid=487): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/1_0_0_0', watcher=None) Received response(xid=487): [] Sending request(xid=488): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/1_0_0_0', version=-1) Received response(xid=488): True Sending request(xid=489): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/0_2_2_0', watcher=None) Received response(xid=489): [] Sending request(xid=490): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/0_2_2_0', version=-1) Received response(xid=490): True Sending request(xid=491): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/2_1_1_0', watcher=None) Received response(xid=491): [] Sending request(xid=492): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/2_1_1_0', version=-1) Received response(xid=492): True Sending request(xid=493): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/2_4_4_0', watcher=None) Received response(xid=493): [] Sending request(xid=494): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/2_4_4_0', version=-1) Received response(xid=494): True Sending request(xid=495): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/0_3_3_0', watcher=None) Received response(xid=495): [] Sending request(xid=496): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/0_3_3_0', version=-1) Received response(xid=496): True Sending request(xid=497): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/7_4_4_0', watcher=None) Received response(xid=497): [] Sending request(xid=498): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/7_4_4_0', version=-1) Received response(xid=498): True Sending request(xid=499): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/6_4_4_0', watcher=None) Received response(xid=499): [] Sending request(xid=500): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/6_4_4_0', version=-1) Received response(xid=500): True Sending request(xid=501): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/7_0_4_1', watcher=None) Received response(xid=501): [] Sending request(xid=502): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/7_0_4_1', version=-1) Received response(xid=502): True Sending request(xid=503): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/1_0_4_1', watcher=None) Received response(xid=503): [] Sending request(xid=504): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/1_0_4_1', version=-1) Received response(xid=504): True Sending request(xid=505): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/7_2_2_0', watcher=None) Received response(xid=505): [] Sending request(xid=506): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/7_2_2_0', version=-1) Received response(xid=506): True Sending request(xid=507): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/9_0_4_1', watcher=None) Received response(xid=507): [] Sending request(xid=508): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/9_0_4_1', version=-1) Received response(xid=508): True Sending request(xid=509): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/8_0_0_0', watcher=None) Received response(xid=509): [] Sending request(xid=510): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/8_0_0_0', version=-1) Received response(xid=510): True Sending request(xid=511): GetChildren(path='/clickhouse/tables/test/replicas/replica1/parts/5_2_2_0', watcher=None) Received response(xid=511): [] Sending request(xid=512): Delete(path='/clickhouse/tables/test/replicas/replica1/parts/5_2_2_0', version=-1) Received response(xid=512): True Sending request(xid=513): Delete(path='/clickhouse/tables/test/replicas/replica1/parts', version=-1) Received response(xid=513): True Sending request(xid=514): GetChildren(path='/clickhouse/tables/test/replicas/replica1/log_pointer', watcher=None) Received response(xid=514): [] Sending request(xid=515): Delete(path='/clickhouse/tables/test/replicas/replica1/log_pointer', version=-1) Received response(xid=515): True Sending request(xid=516): GetChildren(path='/clickhouse/tables/test/replicas/replica1/host', watcher=None) Received response(xid=516): [] Sending request(xid=517): Delete(path='/clickhouse/tables/test/replicas/replica1/host', version=-1) Received response(xid=517): True Sending request(xid=518): GetChildren(path='/clickhouse/tables/test/replicas/replica1/min_unprocessed_insert_time', watcher=None) Received response(xid=518): [] Sending request(xid=519): Delete(path='/clickhouse/tables/test/replicas/replica1/min_unprocessed_insert_time', version=-1) Received response(xid=519): True Sending request(xid=520): GetChildren(path='/clickhouse/tables/test/replicas/replica1/max_processed_insert_time', watcher=None) Received response(xid=520): [] Sending request(xid=521): Delete(path='/clickhouse/tables/test/replicas/replica1/max_processed_insert_time', version=-1) Received response(xid=521): True Sending request(xid=522): GetChildren(path='/clickhouse/tables/test/replicas/replica1/metadata_version', watcher=None) Received response(xid=522): [] Sending request(xid=523): Delete(path='/clickhouse/tables/test/replicas/replica1/metadata_version', version=-1) Received response(xid=523): True Sending request(xid=524): GetChildren(path='/clickhouse/tables/test/replicas/replica1/mutation_pointer', watcher=None) Received response(xid=524): [] Sending request(xid=525): Delete(path='/clickhouse/tables/test/replicas/replica1/mutation_pointer', version=-1) Received response(xid=525): True Sending request(xid=526): GetChildren(path='/clickhouse/tables/test/replicas/replica1/creator_info', watcher=None) Received response(xid=526): [] Sending request(xid=527): Delete(path='/clickhouse/tables/test/replicas/replica1/creator_info', version=-1) Received response(xid=527): True Sending request(xid=528): GetChildren(path='/clickhouse/tables/test/replicas/replica1/is_active', watcher=None) Received response(xid=528): [] Sending request(xid=529): Delete(path='/clickhouse/tables/test/replicas/replica1/is_active', version=-1) Received response(xid=529): True Sending request(xid=530): Delete(path='/clickhouse/tables/test/replicas/replica1', version=-1) Received response(xid=530): True Sending request(xid=531): GetChildren(path='/clickhouse/tables/test/replicas/replica3', watcher=None) Received response(xid=531): ['flags', 'is_lost', 'metadata', 'columns', 'queue', 'parts', 'log_pointer', 'host', 'min_unprocessed_insert_time', 'max_processed_insert_time', 'metadata_version', 'mutation_pointer', 'creator_info', 'is_active'] Sending request(xid=532): GetChildren(path='/clickhouse/tables/test/replicas/replica3/flags', watcher=None) Received response(xid=532): [] Sending request(xid=533): Delete(path='/clickhouse/tables/test/replicas/replica3/flags', version=-1) Received response(xid=533): True Sending request(xid=534): GetChildren(path='/clickhouse/tables/test/replicas/replica3/is_lost', watcher=None) Received response(xid=534): [] Sending request(xid=535): Delete(path='/clickhouse/tables/test/replicas/replica3/is_lost', version=-1) Received response(xid=535): True Sending request(xid=536): GetChildren(path='/clickhouse/tables/test/replicas/replica3/metadata', watcher=None) Received response(xid=536): [] Sending request(xid=537): Delete(path='/clickhouse/tables/test/replicas/replica3/metadata', version=-1) Received response(xid=537): True Sending request(xid=538): GetChildren(path='/clickhouse/tables/test/replicas/replica3/columns', watcher=None) Received response(xid=538): [] Sending request(xid=539): Delete(path='/clickhouse/tables/test/replicas/replica3/columns', version=-1) Received response(xid=539): True Sending request(xid=540): GetChildren(path='/clickhouse/tables/test/replicas/replica3/queue', watcher=None) Received response(xid=540): [] Sending request(xid=541): Delete(path='/clickhouse/tables/test/replicas/replica3/queue', version=-1) Executing query SELECT count() FROM test_log_table on switching_node Received response(xid=541): True Sending request(xid=542): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts', watcher=None) Received response(xid=542): ['3_0_0_0', '3_2_2_0', '5_0_4_1', '4_4_4_0', '3_4_4_0', '7_0_0_0', '9_4_4_0', '3_3_3_0', '6_1_1_0', '5_1_1_0', '7_3_3_0', '1_4_4_0', '1_3_3_0', '4_3_3_0', '9_0_0_0', '8_4_4_0', '9_1_1_0', '7_1_1_0', '4_0_4_1', '4_2_2_0', '2_3_3_0', '6_0_4_1', '9_2_2_0', '3_0_4_1', '1_2_2_0', '6_3_3_0', '0_0_4_1', '6_2_2_0', '0_4_4_0', '4_0_0_0', '6_0_0_0', '0_0_0_0', '5_0_0_0', '1_1_1_0', '3_1_1_0', '8_0_4_1', '5_3_3_0', '4_1_1_0', '2_2_2_0', '0_1_1_0', '9_3_3_0', '5_4_4_0', '8_1_1_0', '2_0_4_1', '2_0_0_0', '8_2_2_0', '8_3_3_0', '1_0_0_0', '0_2_2_0', '2_1_1_0', '2_4_4_0', '0_3_3_0', '7_4_4_0', '6_4_4_0', '7_0_4_1', '1_0_4_1', '7_2_2_0', '9_0_4_1', '8_0_0_0', '5_2_2_0'] Sending request(xid=543): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/3_0_0_0', watcher=None) Received response(xid=543): [] Sending request(xid=544): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/3_0_0_0', version=-1) Received response(xid=544): True Sending request(xid=545): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/3_2_2_0', watcher=None) Received response(xid=545): [] Sending request(xid=546): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/3_2_2_0', version=-1) Received response(xid=546): True Sending request(xid=547): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/5_0_4_1', watcher=None) Received response(xid=547): [] Sending request(xid=548): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/5_0_4_1', version=-1) Received response(xid=548): True Sending request(xid=549): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/4_4_4_0', watcher=None) Received response(xid=549): [] Sending request(xid=550): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/4_4_4_0', version=-1) Received response(xid=550): True Sending request(xid=551): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/3_4_4_0', watcher=None) Received response(xid=551): [] Sending request(xid=552): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/3_4_4_0', version=-1) Received response(xid=552): True Sending request(xid=553): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/7_0_0_0', watcher=None) Received response(xid=553): [] Sending request(xid=554): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/7_0_0_0', version=-1) Received response(xid=554): True Sending request(xid=555): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/9_4_4_0', watcher=None) Received response(xid=555): [] Sending request(xid=556): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/9_4_4_0', version=-1) Received response(xid=556): True Sending request(xid=557): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/3_3_3_0', watcher=None) Received response(xid=557): [] Sending request(xid=558): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/3_3_3_0', version=-1) Received response(xid=558): True Sending request(xid=559): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/6_1_1_0', watcher=None) Received response(xid=559): [] Sending request(xid=560): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/6_1_1_0', version=-1) Received response(xid=560): True Sending request(xid=561): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/5_1_1_0', watcher=None) Received response(xid=561): [] Sending request(xid=562): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/5_1_1_0', version=-1) Received response(xid=562): True Sending request(xid=563): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/7_3_3_0', watcher=None) Received response(xid=563): [] Sending request(xid=564): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/7_3_3_0', version=-1) Received response(xid=564): True Sending request(xid=565): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/1_4_4_0', watcher=None) Received response(xid=565): [] Sending request(xid=566): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/1_4_4_0', version=-1) Received response(xid=566): True Sending request(xid=567): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/1_3_3_0', watcher=None) Received response(xid=567): [] Sending request(xid=568): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/1_3_3_0', version=-1) Received response(xid=568): True Sending request(xid=569): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/4_3_3_0', watcher=None) Received response(xid=569): [] Sending request(xid=570): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/4_3_3_0', version=-1) Received response(xid=570): True Sending request(xid=571): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/9_0_0_0', watcher=None) Received response(xid=571): [] Sending request(xid=572): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/9_0_0_0', version=-1) Received response(xid=572): True Sending request(xid=573): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/8_4_4_0', watcher=None) Received response(xid=573): [] Sending request(xid=574): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/8_4_4_0', version=-1) Received response(xid=574): True Sending request(xid=575): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/9_1_1_0', watcher=None) Received response(xid=575): [] Sending request(xid=576): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/9_1_1_0', version=-1) Received response(xid=576): True Sending request(xid=577): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/7_1_1_0', watcher=None) Received response(xid=577): [] Sending request(xid=578): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/7_1_1_0', version=-1) Received response(xid=578): True Sending request(xid=579): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/4_0_4_1', watcher=None) Received response(xid=579): [] Sending request(xid=580): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/4_0_4_1', version=-1) Received response(xid=580): True Sending request(xid=581): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/4_2_2_0', watcher=None) Received response(xid=581): [] Sending request(xid=582): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/4_2_2_0', version=-1) Received response(xid=582): True Sending request(xid=583): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/2_3_3_0', watcher=None) Received response(xid=583): [] Sending request(xid=584): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/2_3_3_0', version=-1) Received response(xid=584): True Sending request(xid=585): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/6_0_4_1', watcher=None) Received response(xid=585): [] Sending request(xid=586): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/6_0_4_1', version=-1) Received response(xid=586): True Sending request(xid=587): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/9_2_2_0', watcher=None) Received response(xid=587): [] Sending request(xid=588): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/9_2_2_0', version=-1) Received response(xid=588): True Sending request(xid=589): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/3_0_4_1', watcher=None) Received response(xid=589): [] Sending request(xid=590): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/3_0_4_1', version=-1) Received response(xid=590): True Sending request(xid=591): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/1_2_2_0', watcher=None) Received response(xid=591): [] Sending request(xid=592): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/1_2_2_0', version=-1) Received response(xid=592): True Sending request(xid=593): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/6_3_3_0', watcher=None) Received response(xid=593): [] Sending request(xid=594): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/6_3_3_0', version=-1) Received response(xid=594): True Sending request(xid=595): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/0_0_4_1', watcher=None) Received response(xid=595): [] Sending request(xid=596): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/0_0_4_1', version=-1) Received response(xid=596): True Sending request(xid=597): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/6_2_2_0', watcher=None) Received response(xid=597): [] Sending request(xid=598): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/6_2_2_0', version=-1) Received response(xid=598): True Sending request(xid=599): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/0_4_4_0', watcher=None) Received response(xid=599): [] Sending request(xid=600): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/0_4_4_0', version=-1) Received response(xid=600): True Sending request(xid=601): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/4_0_0_0', watcher=None) Received response(xid=601): [] Sending request(xid=602): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/4_0_0_0', version=-1) Received response(xid=602): True Sending request(xid=603): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/6_0_0_0', watcher=None) Received response(xid=603): [] Sending request(xid=604): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/6_0_0_0', version=-1) Starting new HTTP connection (1): 172.16.10.2:9091 Received response(xid=604): True Sending request(xid=605): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/0_0_0_0', watcher=None) Received response(xid=605): [] Sending request(xid=606): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/0_0_0_0', version=-1) Received response(xid=606): True Sending request(xid=607): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/5_0_0_0', watcher=None) Received response(xid=607): [] Sending request(xid=608): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/5_0_0_0', version=-1) Received response(xid=608): True Sending request(xid=609): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/1_1_1_0', watcher=None) Received response(xid=609): [] Sending request(xid=610): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/1_1_1_0', version=-1) Received response(xid=610): True Sending request(xid=611): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/3_1_1_0', watcher=None) Received response(xid=611): [] Sending request(xid=612): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/3_1_1_0', version=-1) Received response(xid=612): True Sending request(xid=613): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/8_0_4_1', watcher=None) Received response(xid=613): [] Sending request(xid=614): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/8_0_4_1', version=-1) Received response(xid=614): True Sending request(xid=615): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/5_3_3_0', watcher=None) Received response(xid=615): [] Sending request(xid=616): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/5_3_3_0', version=-1) Received response(xid=616): True Sending request(xid=617): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/4_1_1_0', watcher=None) Received response(xid=617): [] Sending request(xid=618): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/4_1_1_0', version=-1) Received response(xid=618): True Sending request(xid=619): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/2_2_2_0', watcher=None) Received response(xid=619): [] Sending request(xid=620): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/2_2_2_0', version=-1) http://172.16.10.2:9091 "GET /api/v1/query?query=up&time=1743566308.7599242 HTTP/1.1" 200 161 Received response(xid=620): True Executing query DROP TABLE IF EXISTS prometheus SYNC on node Sending request(xid=621): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/0_1_1_0', watcher=None) [gw3] PASSED test_prometheus_protocols/test.py::test_default Received response(xid=621): [] Sending request(xid=622): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/0_1_1_0', version=-1) Received response(xid=622): True Sending request(xid=623): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/9_3_3_0', watcher=None) Received response(xid=623): [] Sending request(xid=624): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/9_3_3_0', version=-1) Received response(xid=624): True Sending request(xid=625): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/5_4_4_0', watcher=None) Received response(xid=625): [] Sending request(xid=626): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/5_4_4_0', version=-1) Received response(xid=626): True Sending request(xid=627): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/8_1_1_0', watcher=None) Received response(xid=627): [] Sending request(xid=628): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/8_1_1_0', version=-1) Received response(xid=628): True Sending request(xid=629): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/2_0_4_1', watcher=None) Received response(xid=629): [] Sending request(xid=630): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/2_0_4_1', version=-1) Received response(xid=630): True Sending request(xid=631): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/2_0_0_0', watcher=None) Received response(xid=631): [] Sending request(xid=632): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/2_0_0_0', version=-1) Received response(xid=632): True Sending request(xid=633): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/8_2_2_0', watcher=None) Received response(xid=633): [] Sending request(xid=634): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/8_2_2_0', version=-1) Received response(xid=634): True Sending request(xid=635): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/8_3_3_0', watcher=None) Received response(xid=635): [] Sending request(xid=636): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/8_3_3_0', version=-1) Received response(xid=636): True Sending request(xid=637): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/1_0_0_0', watcher=None) Received response(xid=637): [] Sending request(xid=638): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/1_0_0_0', version=-1) Received response(xid=638): True Sending request(xid=639): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/0_2_2_0', watcher=None) Received response(xid=639): [] Sending request(xid=640): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/0_2_2_0', version=-1) Received response(xid=640): True Sending request(xid=641): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/2_1_1_0', watcher=None) Received response(xid=641): [] Sending request(xid=642): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/2_1_1_0', version=-1) Received response(xid=642): True Sending request(xid=643): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/2_4_4_0', watcher=None) Received response(xid=643): [] Sending request(xid=644): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/2_4_4_0', version=-1) Received response(xid=644): True Sending request(xid=645): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/0_3_3_0', watcher=None) Received response(xid=645): [] Sending request(xid=646): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/0_3_3_0', version=-1) Received response(xid=646): True Sending request(xid=647): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/7_4_4_0', watcher=None) Received response(xid=647): [] Sending request(xid=648): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/7_4_4_0', version=-1) Received response(xid=648): True Sending request(xid=649): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/6_4_4_0', watcher=None) Received response(xid=649): [] Sending request(xid=650): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/6_4_4_0', version=-1) Received response(xid=650): True Sending request(xid=651): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/7_0_4_1', watcher=None) Received response(xid=651): [] Sending request(xid=652): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/7_0_4_1', version=-1) run container_id:roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cat /etc/clickhouse-server/config.d/switching_node.xml'] Command:[docker exec roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 bash -c cat /etc/clickhouse-server/config.d/switching_node.xml] Received response(xid=652): True Sending request(xid=653): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/1_0_4_1', watcher=None) Received response(xid=653): [] Sending request(xid=654): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/1_0_4_1', version=-1) Received response(xid=654): True Sending request(xid=655): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/7_2_2_0', watcher=None) Received response(xid=655): [] Sending request(xid=656): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/7_2_2_0', version=-1) Received response(xid=656): True Sending request(xid=657): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/9_0_4_1', watcher=None) Received response(xid=657): [] Sending request(xid=658): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/9_0_4_1', version=-1) Received response(xid=658): True Sending request(xid=659): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/8_0_0_0', watcher=None) Received response(xid=659): [] Sending request(xid=660): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/8_0_0_0', version=-1) Received response(xid=660): True Sending request(xid=661): GetChildren(path='/clickhouse/tables/test/replicas/replica3/parts/5_2_2_0', watcher=None) Received response(xid=661): [] Sending request(xid=662): Delete(path='/clickhouse/tables/test/replicas/replica3/parts/5_2_2_0', version=-1) Received response(xid=662): True Sending request(xid=663): Delete(path='/clickhouse/tables/test/replicas/replica3/parts', version=-1) Received response(xid=663): True Sending request(xid=664): GetChildren(path='/clickhouse/tables/test/replicas/replica3/log_pointer', watcher=None) Received response(xid=664): [] Sending request(xid=665): Delete(path='/clickhouse/tables/test/replicas/replica3/log_pointer', version=-1) Received response(xid=665): True Sending request(xid=666): GetChildren(path='/clickhouse/tables/test/replicas/replica3/host', watcher=None) Received response(xid=666): [] Sending request(xid=667): Delete(path='/clickhouse/tables/test/replicas/replica3/host', version=-1) Received response(xid=667): True Sending request(xid=668): GetChildren(path='/clickhouse/tables/test/replicas/replica3/min_unprocessed_insert_time', watcher=None) Received response(xid=668): [] Sending request(xid=669): Delete(path='/clickhouse/tables/test/replicas/replica3/min_unprocessed_insert_time', version=-1) Received response(xid=669): True Sending request(xid=670): GetChildren(path='/clickhouse/tables/test/replicas/replica3/max_processed_insert_time', watcher=None) Received response(xid=670): [] Sending request(xid=671): Delete(path='/clickhouse/tables/test/replicas/replica3/max_processed_insert_time', version=-1) Received response(xid=671): True Sending request(xid=672): GetChildren(path='/clickhouse/tables/test/replicas/replica3/metadata_version', watcher=None) Received response(xid=672): [] Sending request(xid=673): Delete(path='/clickhouse/tables/test/replicas/replica3/metadata_version', version=-1) Received response(xid=673): True Sending request(xid=674): GetChildren(path='/clickhouse/tables/test/replicas/replica3/mutation_pointer', watcher=None) Received response(xid=674): [] Sending request(xid=675): Delete(path='/clickhouse/tables/test/replicas/replica3/mutation_pointer', version=-1) Received response(xid=675): True Sending request(xid=676): GetChildren(path='/clickhouse/tables/test/replicas/replica3/creator_info', watcher=None) Received response(xid=676): [] Sending request(xid=677): Delete(path='/clickhouse/tables/test/replicas/replica3/creator_info', version=-1) Received response(xid=677): True Sending request(xid=678): GetChildren(path='/clickhouse/tables/test/replicas/replica3/is_active', watcher=None) Received response(xid=678): [] Sending request(xid=679): Delete(path='/clickhouse/tables/test/replicas/replica3/is_active', version=-1) Received response(xid=679): True Sending request(xid=680): Delete(path='/clickhouse/tables/test/replicas/replica3', version=-1) Executing query DROP TABLE IF EXISTS original SYNC on node Received response(xid=680): True Sending request(xid=681): Delete(path='/clickhouse/tables/test/replicas', version=-1) Received response(xid=681): True Sending request(xid=682): GetChildren(path='/clickhouse/tables/test/alter_partition_version', watcher=None) Received response(xid=682): [] Sending request(xid=683): Delete(path='/clickhouse/tables/test/alter_partition_version', version=-1) Received response(xid=683): True Sending request(xid=684): GetChildren(path='/clickhouse/tables/test/leader_election', watcher=None) Received response(xid=684): ['leader_election-0'] Sending request(xid=685): GetChildren(path='/clickhouse/tables/test/leader_election/leader_election-0', watcher=None) Received response(xid=685): [] Sending request(xid=686): Delete(path='/clickhouse/tables/test/leader_election/leader_election-0', version=-1) Received response(xid=686): True Sending request(xid=687): Delete(path='/clickhouse/tables/test/leader_election', version=-1) Received response(xid=687): True Sending request(xid=688): GetChildren(path='/clickhouse/tables/test/nonincrement_block_numbers', watcher=None) Received response(xid=688): [] Sending request(xid=689): Delete(path='/clickhouse/tables/test/nonincrement_block_numbers', version=-1) Received response(xid=689): True Sending request(xid=690): GetChildren(path='/clickhouse/tables/test/columns', watcher=None) Received response(xid=690): [] Sending request(xid=691): Delete(path='/clickhouse/tables/test/columns', version=-1) Stdout: Stdout: Stdout: Stdout: 1 Stdout: run container_id:roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/1010 Stdout: Stdout: Stdout: 0 Stdout: run container_id:roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/0101 /var/lib/clickhouse/disks/s3/store/da9/da9a246d-cb84-4576-9a5f-e310d7402ff2/detached/all_1_1_0/primary.cidx"] Command:[docker exec roottestremoteblobsnamingbackwardcompatibility-gw1-node-1 bash -c echo '5 1 50 50 old-style-prefix/with-several-section/cek/wuhnxlqkiavbvmahggzwjfauymrgf 0 1 ' > /var/lib/clickhouse/disks/s3/store/da9/da9a246d-cb84-4576-9a5f-e310d7402ff2/detached/all_1_1_0/primary.cidx] Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node2 Executing query SELECT count() FROM system.parts WHERE table = 'test_read_new_format' and active on node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query DROP TABLE postgres_database.test_table on node1 Executing query ALTER TABLE test_read_new_format ATTACH PART 'all_1_1_0' on node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SHOW TABLES FROM postgres_database on node1 Executing query SELECT count() FROM system.parts WHERE table = 'test_read_new_format' and active on node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM test_read_new_format on node Executing query ATTACH TABLE postgres_database.test_table on node1 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Starting new HTTP connection (1): 172.16.10.2:9091 http://172.16.10.2:9091 "GET /api/v1/query?query=up&time=1743566322.7973268 HTTP/1.1" 200 161 Executing query DROP TABLE IF EXISTS prometheus SYNC on node [gw3] PASSED test_prometheus_protocols/test.py::test_tags_to_columns run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrefreshablemv-gw5-node1-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/b5d757ce4d9b05907a48d5573bd28047f239e8a0400ee6e7a25d818c118f0278/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/b5d757ce4d9b05907a48d5573bd28047f239e8a0400ee6e7a25d818c118f0278/json HTTP/1.1" 200 586 Executing query DROP TABLE IF EXISTS original SYNC on node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SHOW TABLES FROM postgres_database on node1 Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node2 Executing query SELECT name FROM system.parts WHERE table = 'test_read_new_format' and active LIMIT 1 on node Executing query DETACH TABLE postgres_database.test_table on node1 Executing query SELECT path FROM system.parts WHERE table = 'test_read_new_format' and name = 'all_2_2_0' on node Executing query DROP TABLE IF EXISTS mydata SYNC on node Executing query GRANT SELECT(x10) ON tbl TO R3 on instance Executing query SHOW TABLES FROM postgres_database on node1 Executing query SELECT remote_path FROM system.remote_data_paths WHERE concat(path, local_path) = '/var/lib/clickhouse/disks/s3/store/da9/da9a246d-cb84-4576-9a5f-e310d7402ff2/all_2_2_0/primary.cidx' on node Executing query DROP TABLE IF EXISTS mytable SYNC on node Executing query GRANT R3 TO R2 on instance Stderr: Container roottestreplicationcredentials-gw8-node7-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node3-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node1-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node4-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node2-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node8-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node6-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node5-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node7-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node8-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node5-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node6-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node3-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node4-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node1-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node2-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-zoo3-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-zoo1-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-zoo2-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-zoo1-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-zoo2-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-zoo3-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query ATTACH TABLE postgres_database.test_table on node1 Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] [gw1] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] [gw1] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] Executing query CREATE TABLE test_replicated_merge_tree ( id Int64, val String ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/test_replicated_merge_tree_s3', '{replica}') PARTITION BY id ORDER BY (id, val) SETTINGS storage_policy='s3', allow_remote_fs_zero_copy_replication='0' on node Executing query DROP TABLE IF EXISTS mymetrics SYNC on node Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/.env --project-name roottestreplicationcredentials-gw8 --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/docker-compose.yml down --volumes] Executing query SHOW TABLES FROM postgres_database on node1 Command:[docker compose --env-file /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/.env --project-name roottestprometheusprotocols-gw3 --file /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_prometheus.yml stop --timeout 20] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query DROP DATABASE postgres_database on node1 Executing query CREATE TABLE test_replicated_merge_tree ( id Int64, val String ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/test_replicated_merge_tree_s3', '{replica}') PARTITION BY id ORDER BY (id, val) SETTINGS storage_policy='s3', allow_remote_fs_zero_copy_replication='0' on new_node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SHOW DATABASES on node1 Executing query INSERT INTO test_replicated_merge_tree VALUES (0, 'a') on node [gw2] PASSED test_postgresql_database_engine/test.py::test_postgresql_database_engine_with_clickhouse_ddl test_postgresql_database_engine/test.py::test_postgresql_database_with_schema Executing query CREATE DATABASE postgres_database ENGINE = PostgreSQL('postgres1:5432', 'postgres_database', 'postgres', 'mysecretpassword', 'test_schema') on node1 Executing query GRANT R3 TO C on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node2 Executing query SHOW TABLES FROM postgres_database on node1 Executing query REVOKE SELECT(x10) ON tbl FROM R1, R2, R3 on instance Executing query INSERT INTO test_replicated_merge_tree VALUES (1, 'b') on new_node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query INSERT INTO postgres_database.table1 SELECT number from numbers(10000) on node1 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SYSTEM SYNC REPLICA test_replicated_merge_tree on node Stdout:763 Clickhouse process running. run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:763 Executing query select 20 on node1 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SYSTEM SYNC REPLICA test_replicated_merge_tree on new_node Executing query SELECT count() FROM postgres_database.table1 on node1 Stderr: Container roottestreplicationcredentials-gw8-node5-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node3-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node1-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node7-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node6-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node8-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node2-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node4-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-node5-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node5-1 Removing Stderr: Container roottestreplicationcredentials-gw8-node1-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node1-1 Removing Stderr: Container roottestreplicationcredentials-gw8-node2-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node2-1 Removing Stderr: Container roottestreplicationcredentials-gw8-node3-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node3-1 Removing Stderr: Container roottestreplicationcredentials-gw8-node8-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node8-1 Removing Stderr: Container roottestreplicationcredentials-gw8-node6-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node6-1 Removing Stderr: Container roottestreplicationcredentials-gw8-node4-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node4-1 Removing Stderr: Container roottestreplicationcredentials-gw8-node7-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-node7-1 Removing Stderr: Container roottestreplicationcredentials-gw8-node5-1 Removed Stderr: Container roottestreplicationcredentials-gw8-node2-1 Removed Stderr: Container roottestreplicationcredentials-gw8-node7-1 Removed Stderr: Container roottestreplicationcredentials-gw8-node1-1 Removed Stderr: Container roottestreplicationcredentials-gw8-node8-1 Removed Stderr: Container roottestreplicationcredentials-gw8-node6-1 Removed Stderr: Container roottestreplicationcredentials-gw8-node3-1 Removed Stderr: Container roottestreplicationcredentials-gw8-node4-1 Removed Stderr: Container roottestreplicationcredentials-gw8-zoo2-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-zoo3-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-zoo1-1 Stopping Stderr: Container roottestreplicationcredentials-gw8-zoo3-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-zoo3-1 Removing Stderr: Container roottestreplicationcredentials-gw8-zoo2-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-zoo2-1 Removing Stderr: Container roottestreplicationcredentials-gw8-zoo1-1 Stopped Stderr: Container roottestreplicationcredentials-gw8-zoo1-1 Removing Stderr: Container roottestreplicationcredentials-gw8-zoo1-1 Removed Stderr: Container roottestreplicationcredentials-gw8-zoo3-1 Removed Stderr: Container roottestreplicationcredentials-gw8-zoo2-1 Removed Stderr: Network roottestreplicationcredentials-gw8_default Removing Stderr: Network roottestreplicationcredentials-gw8_default Removed Cleanup called Docker networks for project roottestreplicationcredentials-gw8 are NETWORK ID NAME DRIVER SCOPE Executing query select '===test_refresh_vs_shutdown_smoke start===' on node1 Executing query SELECT count() FROM test_replicated_merge_tree on node Docker containers for project roottestreplicationcredentials-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query DETACH TABLE postgres_database.table1 on node1 Docker volumes for project roottestreplicationcredentials-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicationcredentials-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreplicationcredentials-gw8 Trying to prune unused networks... Executing query GRANT SELECT(x9) ON tbl TO R1 on instance Trying to prune unused images... Command:[docker image prune -f] Executing query create materialized view re.a0 refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select number*10 as x from numbers(2) on node1 Executing query SELECT count() FROM test_replicated_merge_tree on new_node Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query ATTACH TABLE postgres_database.table1 on node1 Stdout:8 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 8 Command:[docker compose --env-file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/.env --project-name roottestreplicationcredentials-gw8 --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/docker-compose.yml stop --timeout 20] Executing query GRANT SELECT(x10) ON tbl TO R2 on instance Executing query DROP TABLE IF EXISTS test_replicated_merge_tree SYNC on node Executing query SELECT count() FROM postgres_database.table1 on node1 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query DROP DATABASE postgres_database on node1 Executing query create materialized view re.a1 refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select number*10 as x from numbers(2) on node1 Executing query DROP TABLE IF EXISTS test_replicated_merge_tree SYNC on new_node Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node2 Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] [gw2] PASSED test_postgresql_database_engine/test.py::test_postgresql_database_with_schema test_postgresql_database_engine/test.py::test_postgresql_fetch_tables Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query CREATE DATABASE postgres_database ENGINE = PostgreSQL('postgres1:5432', 'postgres_database', 'postgres', 'mysecretpassword') on node1 Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/.env --project-name roottestreplicationcredentials-gw8 --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/docker-compose.yml down --volumes] Executing query SELECT default_compression_codec FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node2 Executing query SHOW TABLES FROM postgres_database on node1 [gw1] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] Executing query CREATE TABLE test_replicated_merge_tree ( id Int64, val String ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/test_replicated_merge_tree_s3_zero_copy', '{replica}') PARTITION BY id ORDER BY (id, val) SETTINGS storage_policy='s3', allow_remote_fs_zero_copy_replication='1' on node run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c ps -C clickhouse] Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node2 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance run container_id:roottestpostgresqldatabaseengine-gw2-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "PostgreSQL table table1 does not exist" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestpostgresqldatabaseengine-gw2-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "PostgreSQL table table1 does not exist" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout: PID TTY TIME CMD Stdout: 763 ? 00:00:00 clickhouse run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c pkill clickhouse] [gw2] PASSED test_postgresql_database_engine/test.py::test_postgresql_fetch_tables test_postgresql_database_engine/test.py::test_postgresql_password_leak run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query DROP DATABASE IF EXISTS postgres_database on node1 Executing query CREATE TABLE test_replicated_merge_tree ( id Int64, val String ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/test_replicated_merge_tree_s3_zero_copy', '{replica}') PARTITION BY id ORDER BY (id, val) SETTINGS storage_policy='s3', allow_remote_fs_zero_copy_replication='1' on new_node Executing query GRANT SELECT(x4) ON tbl TO R2 on instance Cleanup called Stdout:763 Docker networks for project roottestreplicationcredentials-gw8 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicationcredentials-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query CREATE DATABASE postgres_database ENGINE = PostgreSQL('postgres1:5432', 'postgres_database', 'postgres', 'mysecretpassword', 'test_schema') on node1 Executing query GRANT SELECT(x3) ON tbl TO R2 on instance Docker volumes for project roottestreplicationcredentials-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicationcredentials-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreplicationcredentials-gw8 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Executing query INSERT INTO test_replicated_merge_tree VALUES (0, 'a') on node Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query DROP DATABASE IF EXISTS postgres_database2 on node1 Stdout:8 Command:[docker volume prune -f] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Stdout:Total reclaimed space: 0B Volumes pruned: 8 Command:[docker compose --env-file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/.env --project-name roottestreplicationcredentials-gw8 --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/docker-compose.yml stop --timeout 20] Executing query CREATE DATABASE postgres_database2 ENGINE = PostgreSQL('postgres1:5432', 'postgres_database', 'postgres', 'mysecretpassword') on node1 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query INSERT INTO test_replicated_merge_tree VALUES (1, 'b') on new_node Executing query SHOW CREATE postgres_database.table1 on node1 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query SHOW CREATE postgres_database2.table2 on node1 Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query SYSTEM SYNC REPLICA test_replicated_merge_tree on node Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/.env --project-name roottestreplicationcredentials-gw8 --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/docker-compose.yml down --volumes] Executing query GRANT SELECT(x5) ON tbl TO R2 on instance Executing query DROP DATABASE postgres_database on node1 Executing query SYSTEM SYNC REPLICA test_replicated_merge_tree on new_node Executing query GRANT SELECT(x1) ON tbl TO R2 on instance Executing query DROP DATABASE postgres_database2 on node1 Executing query SELECT count() FROM test_replicated_merge_tree on node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node2 [gw2] PASSED test_postgresql_database_engine/test.py::test_postgresql_password_leak test_postgresql_database_engine/test.py::test_predefined_connection_configuration Executing query DROP DATABASE IF EXISTS postgres_database on node1 Cleanup called Executing query SELECT count() FROM test_replicated_merge_tree on new_node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Docker networks for project roottestreplicationcredentials-gw8 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicationcredentials-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreplicationcredentials-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicationcredentials-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] Executing query CREATE DATABASE postgres_database ENGINE = PostgreSQL(postgres1) on node1 Unstopped containers: {} No running containers for project: roottestreplicationcredentials-gw8 Trying to prune unused networks... Executing query SELECT uuid FROM system.tables WHERE name = 'test_replicated_merge_tree' on node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:8 Command:[docker volume prune -f] Executing query select create_table_query from system.tables where database ='postgres_database' on node1 Stdout:Total reclaimed space: 0B Volumes pruned: 8 Command:[docker compose --env-file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/.env --project-name roottestreplicationcredentials-gw8 --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/docker-compose.yml stop --timeout 20] Executing query SELECT remote_path FROM system.remote_data_paths WHERE local_path LIKE '%7c63d0d3-1370-457d-a4ef-3e89c1c78f06%' AND local_path NOT LIKE '%format_version.txt%' ORDER BY ALL on node Executing query GRANT SELECT(x2) ON tbl TO R2 on instance Executing query GRANT SELECT(x6) ON tbl TO R1 on instance Executing query SELECT uuid FROM system.tables WHERE name = 'test_replicated_merge_tree' on new_node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query INSERT INTO postgres_database.test_table SELECT number, number from numbers(100) on node1 Executing query SELECT remote_path FROM system.remote_data_paths WHERE local_path LIKE '%246cff4d-96c4-4a13-81fc-885fdfc2fdda%' AND local_path NOT LIKE '%format_version.txt%' ORDER BY ALL on new_node Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query SELECT count() FROM postgres_database.test_table on node1 Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query SELECT name FROM system.parts WHERE table = 'test_replicated_merge_tree' AND active ORDER BY ALL on node Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/.env --project-name roottestreplicationcredentials-gw8 --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node5/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node6/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node7/docker-compose.yml --file /ClickHouse/tests/integration/test_replication_credentials/_instances-0-gw8/node8/docker-compose.yml down --volumes] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query DROP DATABASE IF EXISTS postgres_database on node1 Executing query SELECT value FROM system.zookeeper WHERE path='/clickhouse/tables/test_replicated_merge_tree_s3_zero_copy' and name='table_shared_id' on node Executing query GRANT SELECT(x8) ON tbl TO R1 on instance Executing query CREATE DATABASE postgres_database ENGINE = PostgreSQL(postgres1, schema='test_schema') on node1 Executing query SELECT name FROM system.zookeeper WHERE path='/clickhouse/zero_copy/zero_copy_s3/7c63d0d3-1370-457d-a4ef-3e89c1c78f06/0_0_0_0' ORDER BY ALL on node Cleanup called Executing query DROP ROLE R1 on instance run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query INSERT INTO postgres_database.test_table SELECT number from numbers(200) on node1 Docker networks for project roottestreplicationcredentials-gw8 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicationcredentials-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query SELECT name FROM system.zookeeper WHERE path='/clickhouse/zero_copy/zero_copy_s3/7c63d0d3-1370-457d-a4ef-3e89c1c78f06/0_0_0_0/old-style-prefix_with-several-section_nds_qymhvqzkmnrscgcaorfoezeesxolw' ORDER BY ALL on node Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node2 Stdout:763 Docker volumes for project roottestreplicationcredentials-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicationcredentials-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT count() FROM postgres_database.test_table on node1 Unstopped containers: {} No running containers for project: roottestreplicationcredentials-gw8 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Executing query SELECT name FROM system.zookeeper WHERE path='/clickhouse/zero_copy/zero_copy_s3/7c63d0d3-1370-457d-a4ef-3e89c1c78f06/1_0_0_0' ORDER BY ALL on node Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Stdout:8 Command:[docker volume prune -f] Executing query DROP DATABASE IF EXISTS postgres_database on node1 Stdout:Total reclaimed space: 0B Volumes pruned: 8 test_replicated_zero_copy_projection_mutation/test.py::test_all_projection_files_are_dropped_when_part_is_dropped ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 ENV HOSTNAME e41643a81eb1 ENV SHLVL 0 ENV HOME /root ENV OLDPWD / ENV DOCKER_HELPER_TAG 5dc43a6382f0 ENV PYTHONUNBUFFERED 1 ENV DOCKER_PYTHON_BOTTLE_TAG caad4729259e ENV UBSAN_OPTIONS print_stacktrace=1 ENV PYTEST_ADDOPTS --dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_postgresql_database_engine/test.py::test_datetime test_postgresql_database_engine/test.py::test_get_create_table_query_with_multidim_arrays test_postgresql_database_engine/test.py::test_inaccessible_postgresql_database_engine_filterable_on_system_tables test_postgresql_database_engine/test.py::test_postgres_database_engine_with_postgres_ddl test_postgresql_database_engine/test.py::test_postgres_database_old_syntax test_postgresql_database_engine/test.py::test_postgresql_database_engine_queries test_postgresql_database_engine/test.py::test_postgresql_database_engine_table_cache test_postgresql_database_engine/test.py::test_postgresql_database_engine_with_clickhouse_ddl test_postgresql_database_engine/test.py::test_postgresql_database_with_schema test_postgresql_database_engine/test.py::test_postgresql_fetch_tables test_postgresql_database_engine/test.py::test_postgresql_password_leak test_postgresql_database_engine/test.py::test_predefined_connection_configuration test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order test_prometheus_endpoint/test.py::test_prometheus_endpoint test_prometheus_protocols/test.py::test_64bit_id test_prometheus_protocols/test.py::test_create_as_table test_prometheus_protocols/test.py::test_custom_id_algorithm test_prometheus_protocols/test.py::test_default test_prometheus_protocols/test.py::test_external_tables test_prometheus_protocols/test.py::test_inner_engines test_prometheus_protocols/test.py::test_read_auth test_prometheus_protocols/test.py::test_remote_write_v1_status_code test_prometheus_protocols/test.py::test_tags_to_columns test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_read_only_table/test.py::test_restart_zookeeper test_recompression_ttl/test.py::test_recompression_multiple_ttls test_recompression_ttl/test.py::test_recompression_replicated test_recompression_ttl/test.py::test_recompression_simple test_recovery_time_metric/test.py::test_recovery_time_metric test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db test_refreshable_mv/test.py::test_refreshable_mv_in_system_db test_relative_filepath/test.py::test_filepath test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers test_reload_certificate/test.py::test_ECcert_reload test_reload_certificate/test.py::test_cert_with_pass_phrase test_reload_certificate/test.py::test_chain_reload test_reload_certificate/test.py::test_first_than_second_cert test_reload_clusters_config/test.py::test_add_cluster test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_simple_reload test_reload_clusters_config/test.py::test_update_one_cluster test_reloading_settings_from_users_xml/test.py::test_force_reload test_reloading_settings_from_users_xml/test.py::test_reload_on_timeout test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_enum test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_int test_reloading_settings_from_users_xml/test.py::test_unknown_setting_force_reload test_reloading_settings_from_users_xml/test.py::test_unknown_setting_reload_on_timeout 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain]' test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4]' test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format test_render_log_file_name_templates/test.py::test_check_file_names test_replica_can_become_leader/test.py::test_can_become_leader test_replica_is_active/test.py::test_replica_is_active test_replicated_zero_copy_projection_mutation/test.py::test_all_projection_files_are_dropped_when_part_is_dropped test_replicated_zero_copy_projection_mutation/test.py::test_hardlinks_preserved_when_projection_dropped test_replicating_constants/test.py::test_different_versions test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_restart_server/test.py::test_drop_memory_database test_restart_server/test.py::test_flushes_async_insert_queue test_restore_replica/test.py::test_restore_replica_alive_replicas test_restore_replica/test.py::test_restore_replica_invalid_tables test_restore_replica/test.py::test_restore_replica_parallel test_restore_replica/test.py::test_restore_replica_sequential test_rocksdb_read_only/test.py::test_dirctory_missing_after_stop test_rocksdb_read_only/test.py::test_read_only test_role/test.py::test_admin_option test_role/test.py::test_changing_default_roles_affects_new_sessions_only test_role/test.py::test_combine_privileges test_role/test.py::test_create_role test_role/test.py::test_function_current_roles test_role/test.py::test_grant_role_to_role test_role/test.py::test_introspection test_role/test.py::test_revoke_requires_admin_option 'test_role/test.py::test_role_expiration[False]' 'test_role/test.py::test_role_expiration[True]' test_role/test.py::test_roles_cache test_role/test.py::test_set_role test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable 'test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header]' test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated -vvv -ss ENV CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge ENV COMPOSE_HTTP_TIMEOUT 600 ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ENV DOCKER_KERBERIZED_HADOOP_TAG latest ENV DOCKER_CHANNEL stable ENV DOCKER_CLIENT_TIMEOUT 300 ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e ENV PWD /ClickHouse/tests/integration ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 ENV CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config ENV TZ Etc/UTC ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java ENV DOCKER_BASE_TAG 8b2301119731 ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 ENV LC_CTYPE C.UTF-8 ENV INTEGRATION_TESTS_RUN_ID 0 ENV WORKER_FREE_PORTS 30400 30401 30402 30403 30404 30405 30406 30407 30408 30409 30410 30411 30412 30413 30414 30415 30416 30417 30418 30419 30420 30421 30422 30423 30424 30425 30426 30427 30428 30429 30430 30431 30432 30433 30434 30435 30436 30437 30438 30439 30440 30441 30442 30443 30444 30445 30446 30447 30448 30449 ENV PYTEST_XDIST_TESTRUNUID 63960e6506124119bf68bf57c6473af0 ENV PYTEST_XDIST_WORKER gw8 ENV PYTEST_XDIST_WORKER_COUNT 10 ENV PYTEST_CURRENT_TEST test_replicated_zero_copy_projection_mutation/test.py::test_all_projection_files_are_dropped_when_part_is_dropped (setup) CLUSTER INIT base_config_dir:/clickhouse-config clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Setup Keeper Cluster name: project_name:roottestreplicatedzerocopyprojectionmutation-gw8. Added instance name:node1 tag:8b2301119731 base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/.env', '--project-name', 'roottestreplicatedzerocopyprojectionmutation-gw8', '--file', '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Cluster name: project_name:roottestreplicatedzerocopyprojectionmutation-gw8. Added instance name:node2 tag:8b2301119731 base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/.env', '--project-name', 'roottestreplicatedzerocopyprojectionmutation-gw8', '--file', '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node2/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ Running tests in /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/test.py Cluster start called. is_up=False Executing query SELECT name FROM system.zookeeper WHERE path='/clickhouse/zero_copy/zero_copy_s3/7c63d0d3-1370-457d-a4ef-3e89c1c78f06/1_0_0_0/old-style-prefix_with-several-section_ldd_fuctcocxkfjjfkoazfynqgfsojtdf' ORDER BY ALL on node Docker networks for project roottestreplicatedzerocopyprojectionmutation-gw8 are NETWORK ID NAME DRIVER SCOPE Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Docker containers for project roottestreplicatedzerocopyprojectionmutation-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query CREATE DATABASE postgres_database ENGINE = PostgreSQL(postgres1, 'test_schema') on node1 Docker volumes for project roottestreplicatedzerocopyprojectionmutation-gw8 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestreplicatedzerocopyprojectionmutation-gw8 are NETWORK ID NAME DRIVER SCOPE Executing query DROP TABLE IF EXISTS test_replicated_merge_tree SYNC on node Docker containers for project roottestreplicatedzerocopyprojectionmutation-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreplicatedzerocopyprojectionmutation-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicatedzerocopyprojectionmutation-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] Executing query CREATE DATABASE postgres_database ENGINE = PostgreSQL(postgres2) on node1 Unstopped containers: {} No running containers for project: roottestreplicatedzerocopyprojectionmutation-gw8 Trying to prune unused networks... Executing query DROP TABLE IF EXISTS test_replicated_merge_tree SYNC on new_node Trying to prune unused images... Command:[docker image prune -f] Executing query REVOKE SELECT(x5) ON tbl FROM R2, R3 on instance Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query CREATE DATABASE postgres_database ENGINE = PostgreSQL(unknown_collection) on node1 Stdout:8 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 8 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node1/database Setup logs dir /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node1/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node2/database Setup logs dir /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node2/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper3/coordination', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/minio/certs/public.crt', 'RESOLVER_LOGS': '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/resolver', 'RESOLVER_LOGS_FS': 'bind'} stored in /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/.env --project-name roottestreplicatedzerocopyprojectionmutation-gw8 --file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node2/docker-compose.yml pull] Executing query GRANT SELECT(x7) ON tbl TO R2 on instance Executing query CREATE DATABASE postgres_database ENGINE = PostgreSQL(postgres3, port=5432) on node1 [gw1] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] Executing query CREATE TABLE test_replicated_merge_tree ( id Int64, val String ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/test_replicated_merge_tree_s3_template_key', '{replica}') PARTITION BY id ORDER BY (id, val) SETTINGS storage_policy='s3_template_key', allow_remote_fs_zero_copy_replication='0' on node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT count() FROM postgres_database.test_table on node1 Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query DROP DATABASE postgres_database; CREATE DATABASE postgres_database ENGINE = PostgreSQL(postgres1, use_table_cache=1); on node1 Executing query CREATE TABLE test_replicated_merge_tree ( id Int64, val String ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/test_replicated_merge_tree_s3_template_key', '{replica}') PARTITION BY id ORDER BY (id, val) SETTINGS storage_policy='s3_template_key', allow_remote_fs_zero_copy_replication='0' on new_node Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node2 Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SELECT count() FROM postgres_database.test_table on node1 Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Stderr: Container roottestprometheusprotocols-gw3-prometheus_reader-1 Stopping Stderr: Container roottestprometheusprotocols-gw3-prometheus_writer-1 Stopping Stderr: Container roottestprometheusprotocols-gw3-node-1 Stopping Stderr: Container roottestprometheusprotocols-gw3-prometheus_reader-1 Stopped Stderr: Container roottestprometheusprotocols-gw3-prometheus_writer-1 Stopped Stderr: Container roottestprometheusprotocols-gw3-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/.env --project-name roottestprometheusprotocols-gw3 --file /ClickHouse/tests/integration/test_prometheus_protocols/_instances-0-gw3/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_prometheus.yml down --volumes] Executing query INSERT INTO test_replicated_merge_tree VALUES (0, 'a') on node run container_id:roottestpostgresqldatabaseengine-gw2-node1-1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "Cached table `test_table`" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] Command:[docker exec roottestpostgresqldatabaseengine-gw2-node1-1 bash -c [ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "Cached table `test_table`" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query INSERT INTO test_replicated_merge_tree VALUES (1, 'b') on new_node Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 03:58:42.108101 [ 11 ] {c0a25589-3443-4854-b490-927729950fb0} DatabasePostgreSQL(postgres_database): Cached table `test_table` Connection dropped: socket connection error: Connection refused Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 03:58:43.022571 [ 11 ] {83eebdb0-6d81-4f3b-bc01-f9d1a2cee9e9} DatabasePostgreSQL(postgres_database): Cached table `test_table` Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 03:58:43.501266 [ 11 ] {b57668f9-c783-4609-a8fb-751c25e3b064} DatabasePostgreSQL(postgres_database): Cached table `test_table` Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 03:58:43.813355 [ 11 ] {bb4c760a-db6e-4b5b-9560-d1321b4f689d} DatabasePostgreSQL(postgres_database): Cached table `test_table` Stdout:/var/log/clickhouse-server/clickhouse-server.log:2025.04.02 03:58:48.321212 [ 11 ] {662d63bf-1770-4c15-b4fb-691efd360118} DatabasePostgreSQL(postgres_database): Cached table `test_table` Stderr:bash: line 1: test_table: command not found Executing query DROP DATABASE postgres_database on node1 Executing query GRANT SELECT(x5) ON tbl TO R3 on instance Executing query SYSTEM SYNC REPLICA test_replicated_merge_tree on node Command:[docker compose --env-file /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/.env --project-name roottestpostgresqldatabaseengine-gw2 --file /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_postgres.yml stop --timeout 20] [gw2] PASSED test_postgresql_database_engine/test.py::test_predefined_connection_configuration Executing query GRANT SELECT(x8) ON tbl TO R2 on instance Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SYSTEM SYNC REPLICA test_replicated_merge_tree on new_node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT count() FROM test_replicated_merge_tree on node run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Stdout:763 Executing query SELECT count() FROM test_replicated_merge_tree on new_node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query DROP TABLE IF EXISTS test_replicated_merge_tree SYNC on node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node2 Executing query CREATE ROLE R1 on instance Stderr: Container roottestprometheusprotocols-gw3-node-1 Stopping Stderr: Container roottestprometheusprotocols-gw3-prometheus_reader-1 Stopping Stderr: Container roottestprometheusprotocols-gw3-prometheus_writer-1 Stopping Stderr: Container roottestprometheusprotocols-gw3-prometheus_reader-1 Stopped Stderr: Container roottestprometheusprotocols-gw3-prometheus_reader-1 Removing Stderr: Container roottestprometheusprotocols-gw3-node-1 Stopped Stderr: Container roottestprometheusprotocols-gw3-node-1 Removing Stderr: Container roottestprometheusprotocols-gw3-prometheus_writer-1 Stopped Stderr: Container roottestprometheusprotocols-gw3-prometheus_writer-1 Removing Stderr: Container roottestprometheusprotocols-gw3-prometheus_writer-1 Removed Stderr: Container roottestprometheusprotocols-gw3-prometheus_reader-1 Removed Stderr: Container roottestprometheusprotocols-gw3-node-1 Removed Stderr: Network roottestprometheusprotocols-gw3_default Removing Stderr: Network roottestprometheusprotocols-gw3_default Removed Cleanup called Executing query DROP TABLE IF EXISTS test_replicated_merge_tree SYNC on new_node Docker networks for project roottestprometheusprotocols-gw3 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestprometheusprotocols-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query GRANT R1 TO A on instance Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Docker volumes for project roottestprometheusprotocols-gw3 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestprometheusprotocols-gw3-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestprometheusprotocols-gw3 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Executing query GRANT SELECT(x9) ON tbl TO R1 on instance Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] [gw1] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] Executing query CREATE TABLE test_replicated_merge_tree ( id Int64, val String ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/test_replicated_merge_tree_s3_template_key_zero_copy', '{replica}') PARTITION BY id ORDER BY (id, val) SETTINGS storage_policy='s3_template_key', allow_remote_fs_zero_copy_replication='1' on node Stdout:6 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 6 test_rocksdb_read_only/test.py::test_dirctory_missing_after_stop Running tests in /ClickHouse/tests/integration/test_rocksdb_read_only/test.py Cluster start called. is_up=False Docker networks for project roottestrocksdbreadonly-gw3 are NETWORK ID NAME DRIVER SCOPE Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Docker containers for project roottestrocksdbreadonly-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Docker volumes for project roottestrocksdbreadonly-gw3 are DRIVER VOLUME NAME Cleanup called Executing query CREATE TABLE test_replicated_merge_tree ( id Int64, val String ) ENGINE=ReplicatedMergeTree('/clickhouse/tables/test_replicated_merge_tree_s3_template_key_zero_copy', '{replica}') PARTITION BY id ORDER BY (id, val) SETTINGS storage_policy='s3_template_key', allow_remote_fs_zero_copy_replication='1' on new_node Docker networks for project roottestrocksdbreadonly-gw3 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrocksdbreadonly-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrocksdbreadonly-gw3 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrocksdbreadonly-gw3-.*-1$' --format '{{.ID}}:{{.Names}}'] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Unstopped containers: {} No running containers for project: roottestrocksdbreadonly-gw3 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query INSERT INTO test_replicated_merge_tree VALUES (0, 'a') on node Stdout:6 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 6 Setup directory for instance: node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_rocksdb_read_only/configs/rocksdb.xml'] to /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/node/database Setup logs dir /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/node/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/.env --project-name roottestrocksdbreadonly-gw3 --file /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/node/docker-compose.yml pull] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query INSERT INTO test_replicated_merge_tree VALUES (1, 'b') on new_node Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node2 Executing query GRANT SELECT(x6) ON tbl TO R2 on instance Executing query SYSTEM SYNC REPLICA test_replicated_merge_tree on node Executing query REVOKE SELECT(x3) ON tbl FROM R2, R3, R1 on instance Executing query SYSTEM SYNC REPLICA test_replicated_merge_tree on new_node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT count() FROM test_replicated_merge_tree on node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT count() FROM test_replicated_merge_tree on new_node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Stdout:763 Executing query SELECT uuid FROM system.tables WHERE name = 'test_replicated_merge_tree' on node Executing query GRANT SELECT(x3) ON tbl TO R1 on instance Executing query SELECT remote_path FROM system.remote_data_paths WHERE local_path LIKE '%120ed51a-ec49-438b-88de-37784c787009%' AND local_path NOT LIKE '%format_version.txt%' ORDER BY ALL on node Executing query REVOKE SELECT(x2) ON tbl FROM R2, R3, R1 on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT uuid FROM system.tables WHERE name = 'test_replicated_merge_tree' on new_node Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT remote_path FROM system.remote_data_paths WHERE local_path LIKE '%cd713a8f-067e-468c-9e3d-6dd26163f550%' AND local_path NOT LIKE '%format_version.txt%' ORDER BY ALL on new_node Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node2 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.parts WHERE table = 'test_replicated_merge_tree' AND active ORDER BY ALL on node Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query REVOKE SELECT(x8) ON tbl FROM R2, R3, R1 on instance Executing query SELECT value FROM system.zookeeper WHERE path='/clickhouse/tables/test_replicated_merge_tree_s3_template_key_zero_copy' and name='table_shared_id' on node Executing query REVOKE SELECT(x1) ON tbl FROM R2, R3, R1 on instance Executing query SELECT name FROM system.zookeeper WHERE path='/clickhouse/zero_copy/zero_copy_s3/120ed51a-ec49-438b-88de-37784c787009/0_0_0_0' ORDER BY ALL on node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.zookeeper WHERE path='/clickhouse/zero_copy/zero_copy_s3/120ed51a-ec49-438b-88de-37784c787009/0_0_0_0/old-style-prefix_with-several-section_jqe_smezihrqikfdgmwjcoqtxjrxsfpfx' ORDER BY ALL on node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.zookeeper WHERE path='/clickhouse/zero_copy/zero_copy_s3/120ed51a-ec49-438b-88de-37784c787009/1_0_0_0' ORDER BY ALL on node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.zookeeper WHERE path='/clickhouse/zero_copy/zero_copy_s3/120ed51a-ec49-438b-88de-37784c787009/1_0_0_0/prv-first-random-part_new-style-prefix_constant-part_qny_uhekcunvhngmmvzcaafgmogydwqvd' ORDER BY ALL on node Executing query REVOKE SELECT(x10) ON tbl FROM R2, R3, R1 on instance Executing query DROP TABLE IF EXISTS test_replicated_merge_tree SYNC on node Executing query GRANT SELECT(x1) ON tbl TO R1 on instance Executing query DROP TABLE IF EXISTS test_replicated_merge_tree SYNC on new_node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node2 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query CREATE TABLE test_read_new_format ( id Int64, data String ) ENGINE=MergeTree() ORDER BY id on new_node [gw1] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query GRANT SELECT(x10) ON tbl TO R3 on instance Executing query INSERT INTO test_read_new_format VALUES (1, 'Hello') on new_node Stdout:763 Executing query GRANT SELECT(x8) ON tbl TO R3 on instance Executing query SELECT name FROM system.parts WHERE table = 'test_read_new_format' and active LIMIT 1 on new_node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT path FROM system.parts WHERE table = 'test_read_new_format' and name = 'all_1_1_0' on new_node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT remote_path FROM system.remote_data_paths WHERE concat(path, local_path) = '/var/lib/clickhouse/disks/s3/store/7b5/7b5ad298-4320-4a78-803c-454c3ecf4660/all_1_1_0/primary.cidx' on new_node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node2 Executing query SELECT default_compression_codec FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node2 Executing query ALTER TABLE test_read_new_format DETACH PART 'all_1_1_0' on new_node Executing query GRANT SELECT(x2) ON tbl TO R1 on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_3' and table = 'table_for_recompression' on node2 run container_id:roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1 detach:False nothrow:False cmd: ['bash', '-c', 'cat /var/lib/clickhouse/disks/s3/store/7b5/7b5ad298-4320-4a78-803c-454c3ecf4660/detached/all_1_1_0/primary.cidx'] Command:[docker exec roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1 bash -c cat /var/lib/clickhouse/disks/s3/store/7b5/7b5ad298-4320-4a78-803c-454c3ecf4660/detached/all_1_1_0/primary.cidx] Stderr: Container roottests3cluster-gw4-resolver-1 Stopping Stderr: Container roottests3cluster-gw4-s0_0_0-1 Stopping Stderr: Container roottests3cluster-gw4-s0_1_0-1 Stopping Stderr: Container roottests3cluster-gw4-s0_0_1-1 Stopping Stderr: Container roottests3cluster-gw4-s0_0_1-1 Stopped Stderr: Container roottests3cluster-gw4-s0_0_0-1 Stopped Stderr: Container roottests3cluster-gw4-minio1-1 Stopping Stderr: Container roottests3cluster-gw4-minio1-1 Stopped Stderr: Container roottests3cluster-gw4-s0_1_0-1 Stopped Stderr: Container roottests3cluster-gw4-zoo1-1 Stopping Stderr: Container roottests3cluster-gw4-zoo2-1 Stopping Stderr: Container roottests3cluster-gw4-zoo3-1 Stopping Stderr: Container roottests3cluster-gw4-zoo1-1 Stopped Stderr: Container roottests3cluster-gw4-zoo2-1 Stopped Stderr: Container roottests3cluster-gw4-zoo3-1 Stopped Stderr: Container roottests3cluster-gw4-resolver-1 Stopped Stderr: Container roottests3cluster-gw4-proxy2-1 Stopping Stderr: Container roottests3cluster-gw4-proxy1-1 Stopping Stderr: Container roottests3cluster-gw4-proxy1-1 Stopped Stderr: Container roottests3cluster-gw4-proxy2-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query REVOKE SELECT(x5) ON tbl FROM R2, R3, R1 on instance Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_1_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_1_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:5 Stdout:1 50 Stdout:50 old-style-prefix/with-several-section/qup/llbwlfnvpivnwtyrfkixyyrorkpfp Stdout:0 Stdout:1 Stdout: Command:[docker compose --env-file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/.env --project-name roottestremoteblobsnamingbackwardcompatibility-gw1 --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/new_node/docker-compose.yml --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/switching_node/docker-compose.yml stop --timeout 20] [gw1] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/.env --project-name roottests3cluster-gw4 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/s0_1_0/docker-compose.yml down --volumes] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Connection dropped: outstanding heartbeat ping not received Transition to CONNECTING Zookeeper connection lost run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Stdout:763 Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Executing query SELECT name FROM system.parts where name = 'all_1_1_3' and table = 'table_for_recompression' on node2 Stderr: Container roottestpostgresqldatabaseengine-gw2-node1-1 Stopping Stderr: Container roottestpostgresqldatabaseengine-gw2-postgres1-1 Stopping Stderr: Container roottestpostgresqldatabaseengine-gw2-postgres1-1 Stopped Stderr: Container roottestpostgresqldatabaseengine-gw2-node1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/.env --project-name roottestpostgresqldatabaseengine-gw2 --file /ClickHouse/tests/integration/test_postgresql_database_engine/_instances-0-gw2/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_postgres.yml down --volumes] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Stderr: Container roottests3cluster-gw4-s0_1_0-1 Stopping Stderr: Container roottests3cluster-gw4-s0_0_1-1 Stopping Stderr: Container roottests3cluster-gw4-s0_0_0-1 Stopping Stderr: Container roottests3cluster-gw4-resolver-1 Stopping Stderr: Container roottests3cluster-gw4-resolver-1 Stopped Stderr: Container roottests3cluster-gw4-resolver-1 Removing Stderr: Container roottests3cluster-gw4-s0_0_1-1 Stopped Stderr: Container roottests3cluster-gw4-s0_0_1-1 Removing Stderr: Container roottests3cluster-gw4-s0_1_0-1 Stopped Stderr: Container roottests3cluster-gw4-s0_1_0-1 Removing Stderr: Container roottests3cluster-gw4-s0_0_0-1 Stopped Stderr: Container roottests3cluster-gw4-s0_0_0-1 Removing Stderr: Container roottests3cluster-gw4-s0_0_0-1 Removed Stderr: Container roottests3cluster-gw4-minio1-1 Stopping Stderr: Container roottests3cluster-gw4-minio1-1 Stopped Stderr: Container roottests3cluster-gw4-minio1-1 Removing Stderr: Container roottests3cluster-gw4-resolver-1 Removed Stderr: Container roottests3cluster-gw4-s0_1_0-1 Removed Stderr: Container roottests3cluster-gw4-s0_0_1-1 Removed Stderr: Container roottests3cluster-gw4-zoo3-1 Stopping Stderr: Container roottests3cluster-gw4-zoo1-1 Stopping Stderr: Container roottests3cluster-gw4-zoo2-1 Stopping Stderr: Container roottests3cluster-gw4-zoo2-1 Stopped Stderr: Container roottests3cluster-gw4-zoo2-1 Removing Stderr: Container roottests3cluster-gw4-zoo3-1 Stopped Stderr: Container roottests3cluster-gw4-zoo3-1 Removing Stderr: Container roottests3cluster-gw4-zoo1-1 Stopped Stderr: Container roottests3cluster-gw4-zoo1-1 Removing Stderr: Container roottests3cluster-gw4-zoo3-1 Removed Stderr: Container roottests3cluster-gw4-zoo2-1 Removed Stderr: Container roottests3cluster-gw4-zoo1-1 Removed Stderr: Container roottests3cluster-gw4-minio1-1 Removed Stderr: Container roottests3cluster-gw4-proxy1-1 Stopping Stderr: Container roottests3cluster-gw4-proxy2-1 Stopping Stderr: Container roottests3cluster-gw4-proxy1-1 Stopped Stderr: Container roottests3cluster-gw4-proxy1-1 Removing Stderr: Container roottests3cluster-gw4-proxy2-1 Stopped Stderr: Container roottests3cluster-gw4-proxy2-1 Removing Stderr: Container roottests3cluster-gw4-proxy1-1 Removed Stderr: Container roottests3cluster-gw4-proxy2-1 Removed Stderr: Volume roottests3cluster-gw4_data1-1 Removing Stderr: Network roottests3cluster-gw4_default Removing Stderr: Volume roottests3cluster-gw4_data1-1 Removed Stderr: Network roottests3cluster-gw4_default Removed Cleanup called Docker networks for project roottests3cluster-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottests3cluster-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottests3cluster-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottests3cluster-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Executing query GRANT SELECT(x5) ON tbl TO R3 on instance Unstopped containers: {} No running containers for project: roottests3cluster-gw4 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:3 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 3 test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header] ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 ENV HOSTNAME e41643a81eb1 ENV SHLVL 0 ENV HOME /root ENV OLDPWD / ENV DOCKER_HELPER_TAG 5dc43a6382f0 ENV PYTHONUNBUFFERED 1 ENV DOCKER_PYTHON_BOTTLE_TAG caad4729259e ENV UBSAN_OPTIONS print_stacktrace=1 ENV PYTEST_ADDOPTS --dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_postgresql_database_engine/test.py::test_datetime test_postgresql_database_engine/test.py::test_get_create_table_query_with_multidim_arrays test_postgresql_database_engine/test.py::test_inaccessible_postgresql_database_engine_filterable_on_system_tables test_postgresql_database_engine/test.py::test_postgres_database_engine_with_postgres_ddl test_postgresql_database_engine/test.py::test_postgres_database_old_syntax test_postgresql_database_engine/test.py::test_postgresql_database_engine_queries test_postgresql_database_engine/test.py::test_postgresql_database_engine_table_cache test_postgresql_database_engine/test.py::test_postgresql_database_engine_with_clickhouse_ddl test_postgresql_database_engine/test.py::test_postgresql_database_with_schema test_postgresql_database_engine/test.py::test_postgresql_fetch_tables test_postgresql_database_engine/test.py::test_postgresql_password_leak test_postgresql_database_engine/test.py::test_predefined_connection_configuration test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order test_prometheus_endpoint/test.py::test_prometheus_endpoint test_prometheus_protocols/test.py::test_64bit_id test_prometheus_protocols/test.py::test_create_as_table test_prometheus_protocols/test.py::test_custom_id_algorithm test_prometheus_protocols/test.py::test_default test_prometheus_protocols/test.py::test_external_tables test_prometheus_protocols/test.py::test_inner_engines test_prometheus_protocols/test.py::test_read_auth test_prometheus_protocols/test.py::test_remote_write_v1_status_code test_prometheus_protocols/test.py::test_tags_to_columns test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_read_only_table/test.py::test_restart_zookeeper test_recompression_ttl/test.py::test_recompression_multiple_ttls test_recompression_ttl/test.py::test_recompression_replicated test_recompression_ttl/test.py::test_recompression_simple test_recovery_time_metric/test.py::test_recovery_time_metric test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db test_refreshable_mv/test.py::test_refreshable_mv_in_system_db test_relative_filepath/test.py::test_filepath test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers test_reload_certificate/test.py::test_ECcert_reload test_reload_certificate/test.py::test_cert_with_pass_phrase test_reload_certificate/test.py::test_chain_reload test_reload_certificate/test.py::test_first_than_second_cert test_reload_clusters_config/test.py::test_add_cluster test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_simple_reload test_reload_clusters_config/test.py::test_update_one_cluster test_reloading_settings_from_users_xml/test.py::test_force_reload test_reloading_settings_from_users_xml/test.py::test_reload_on_timeout test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_enum test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_int test_reloading_settings_from_users_xml/test.py::test_unknown_setting_force_reload test_reloading_settings_from_users_xml/test.py::test_unknown_setting_reload_on_timeout 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain]' test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4]' test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format test_render_log_file_name_templates/test.py::test_check_file_names test_replica_can_become_leader/test.py::test_can_become_leader test_replica_is_active/test.py::test_replica_is_active test_replicated_zero_copy_projection_mutation/test.py::test_all_projection_files_are_dropped_when_part_is_dropped test_replicated_zero_copy_projection_mutation/test.py::test_hardlinks_preserved_when_projection_dropped test_replicating_constants/test.py::test_different_versions test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_restart_server/test.py::test_drop_memory_database test_restart_server/test.py::test_flushes_async_insert_queue test_restore_replica/test.py::test_restore_replica_alive_replicas test_restore_replica/test.py::test_restore_replica_invalid_tables test_restore_replica/test.py::test_restore_replica_parallel test_restore_replica/test.py::test_restore_replica_sequential test_rocksdb_read_only/test.py::test_dirctory_missing_after_stop test_rocksdb_read_only/test.py::test_read_only test_role/test.py::test_admin_option test_role/test.py::test_changing_default_roles_affects_new_sessions_only test_role/test.py::test_combine_privileges test_role/test.py::test_create_role test_role/test.py::test_function_current_roles test_role/test.py::test_grant_role_to_role test_role/test.py::test_introspection test_role/test.py::test_revoke_requires_admin_option 'test_role/test.py::test_role_expiration[False]' 'test_role/test.py::test_role_expiration[True]' test_role/test.py::test_roles_cache test_role/test.py::test_set_role test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable 'test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header]' test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated -vvv -ss ENV CLICKHOUSE_LIBRARY_BRIDGE_BINARY_PATH /clickhouse-library-bridge ENV COMPOSE_HTTP_TIMEOUT 600 ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ENV DOCKER_KERBERIZED_HADOOP_TAG latest ENV DOCKER_CHANNEL stable ENV DOCKER_CLIENT_TIMEOUT 300 ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e ENV PWD /ClickHouse/tests/integration ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 ENV CLICKHOUSE_ODBC_BRIDGE_BINARY_PATH /clickhouse-odbc-bridge ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config ENV TZ Etc/UTC ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java ENV DOCKER_BASE_TAG 8b2301119731 ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 ENV LC_CTYPE C.UTF-8 ENV INTEGRATION_TESTS_RUN_ID 0 ENV WORKER_FREE_PORTS 30200 30201 30202 30203 30204 30205 30206 30207 30208 30209 30210 30211 30212 30213 30214 30215 30216 30217 30218 30219 30220 30221 30222 30223 30224 30225 30226 30227 30228 30229 30230 30231 30232 30233 30234 30235 30236 30237 30238 30239 30240 30241 30242 30243 30244 30245 30246 30247 30248 30249 ENV PYTEST_XDIST_TESTRUNUID 63960e6506124119bf68bf57c6473af0 ENV PYTEST_XDIST_WORKER gw4 ENV PYTEST_XDIST_WORKER_COUNT 10 ENV SSL_CERT_FILE /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw4/minio/certs/public.crt ENV PYTEST_CURRENT_TEST test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header] (setup) CLUSTER INIT base_config_dir:/clickhouse-config Executing query DROP ROLE R1 on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Picked up JAVA_TOOL_OPTIONS: -Djdk.attach.allowAttachSelf=true Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query CREATE ROLE R1 on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_3' and table = 'table_for_recompression' on node2 Executing query GRANT R1 TO A on instance Executing query REVOKE SELECT(x5) ON tbl FROM R2, R3, R1 on instance Stderr: Container roottestpostgresqldatabaseengine-gw2-node1-1 Stopping Stderr: Container roottestpostgresqldatabaseengine-gw2-postgres1-1 Stopping Stderr: Container roottestpostgresqldatabaseengine-gw2-node1-1 Stopped Stderr: Container roottestpostgresqldatabaseengine-gw2-node1-1 Removing Stderr: Container roottestpostgresqldatabaseengine-gw2-postgres1-1 Stopped Stderr: Container roottestpostgresqldatabaseengine-gw2-postgres1-1 Removing Stderr: Container roottestpostgresqldatabaseengine-gw2-node1-1 Removed Stderr: Container roottestpostgresqldatabaseengine-gw2-postgres1-1 Removed Stderr: Network roottestpostgresqldatabaseengine-gw2_default Removing Stderr: Network roottestpostgresqldatabaseengine-gw2_default Removed Cleanup called Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Docker networks for project roottestpostgresqldatabaseengine-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestpostgresqldatabaseengine-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Picked up JAVA_TOOL_OPTIONS: -Djdk.attach.allowAttachSelf=true Stdout:763 Docker volumes for project roottestpostgresqldatabaseengine-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestpostgresqldatabaseengine-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Unstopped containers: {} No running containers for project: roottestpostgresqldatabaseengine-gw2 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:3 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 3 test_prometheus_endpoint/test.py::test_prometheus_endpoint Running tests in /ClickHouse/tests/integration/test_prometheus_endpoint/test.py Cluster start called. is_up=False Docker networks for project roottestprometheusendpoint-gw2 are NETWORK ID NAME DRIVER SCOPE Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Docker containers for project roottestprometheusendpoint-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestprometheusendpoint-gw2 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestprometheusendpoint-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestprometheusendpoint-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestprometheusendpoint-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestprometheusendpoint-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestprometheusendpoint-gw2 Trying to prune unused networks... Executing query GRANT SELECT(x1) ON tbl TO R3 on instance Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:3 Command:[docker volume prune -f] Executing query GRANT SELECT(x5) ON tbl TO R2 on instance Stdout:Total reclaimed space: 0B Volumes pruned: 3 Setup directory for instance: node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_prometheus_endpoint/configs/prom_conf.xml'] to /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/node/database Setup logs dir /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/node/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Executing query SELECT name FROM system.parts where name = 'all_1_1_3' and table = 'table_for_recompression' on node2 http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/.env --project-name roottestprometheusendpoint-gw2 --file /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/node/docker-compose.yml pull] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query GRANT SELECT(x3) ON tbl TO R1 on instance Executing query GRANT SELECT(x2) ON tbl TO R3 on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_3' and table = 'table_for_recompression' on node2 Executing query GRANT SELECT(x9) ON tbl TO R3 on instance run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query DROP ROLE R1 on instance Stdout:763 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_3' and table = 'table_for_recompression' on node2 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query CREATE ROLE R1 on instance Executing query GRANT R1 TO A on instance Executing query GRANT SELECT(x3) ON tbl TO R3 on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance GatewayClient.address is deprecated and will be removed in version 1.0. Use GatewayParameters instead. Command to send: A dfbef6cc0ef2972b9e9510f4805662b425a472689b5e93b01940c781834f8340 Answer received: !yv Command to send: j i rj org.apache.spark.SparkConf e Answer received: !yv Command to send: j i rj org.apache.spark.api.java.* e Answer received: !yv Command to send: j i rj org.apache.spark.api.python.* e Answer received: !yv Command to send: j i rj org.apache.spark.ml.python.* e Answer received: !yv Command to send: j i rj org.apache.spark.mllib.api.python.* e Answer received: !yv Command to send: j i rj org.apache.spark.resource.* e Answer received: !yv Command to send: j i rj org.apache.spark.sql.* e Answer received: !yv Command to send: j i rj org.apache.spark.sql.api.python.* e Answer received: !yv Command to send: j i rj org.apache.spark.sql.hive.* e Answer received: !yv Command to send: j i rj scala.Tuple2 e Answer received: !yv Command to send: r u SparkConf rj e Answer received: !ycorg.apache.spark.SparkConf Command to send: i org.apache.spark.SparkConf bTrue e Answer received: !yro0 Command to send: c o0 set sspark.app.name sspark_test e Answer received: !yro1 Command to send: c o0 set sspark.master slocal e Answer received: !yro2 Command to send: c o0 contains sspark.serializer.objectStreamReset e Answer received: !ybfalse Command to send: c o0 set sspark.serializer.objectStreamReset s100 e Answer received: !yro3 Command to send: c o0 contains sspark.rdd.compress e Answer received: !ybfalse Command to send: c o0 set sspark.rdd.compress sTrue e Answer received: !yro4 Command to send: c o0 contains sspark.master e Answer received: !ybtrue Command to send: c o0 contains sspark.app.name e Answer received: !ybtrue Command to send: c o0 contains sspark.master e Answer received: !ybtrue Command to send: c o0 get sspark.master e Answer received: !yslocal Command to send: c o0 contains sspark.app.name e Answer received: !ybtrue Command to send: c o0 get sspark.app.name e Answer received: !ysspark_test Command to send: c o0 contains sspark.home e Answer received: !ybfalse Command to send: c o0 getAll e Answer received: !yto5 Command to send: a e o5 e Answer received: !yi8 Command to send: a g o5 i0 e Answer received: !yro6 Command to send: c o6 _1 e Answer received: !ysspark.master Command to send: c o6 _2 e Answer received: !yslocal Command to send: a e o5 e Answer received: !yi8 Command to send: a g o5 i1 e Answer received: !yro7 Command to send: c o7 _1 e Answer received: !ysspark.app.submitTime Command to send: c o7 _2 e Answer received: !ys1743566334766 Command to send: a e o5 e Answer received: !yi8 Command to send: a g o5 i2 e Answer received: !yro8 Command to send: c o8 _1 e Answer received: !ysspark.app.name Command to send: c o8 _2 e Answer received: !ysspark_test Command to send: a e o5 e Answer received: !yi8 Command to send: a g o5 i3 e Answer received: !yro9 Command to send: c o9 _1 e Answer received: !ysspark.rdd.compress Command to send: c o9 _2 e Answer received: !ysTrue Command to send: a e o5 e Answer received: !yi8 Command to send: a g o5 i4 e Answer received: !yro10 Command to send: c o10 _1 e Answer received: !ysspark.serializer.objectStreamReset Command to send: c o10 _2 e Answer received: !ys100 Command to send: a e o5 e Answer received: !yi8 Command to send: a g o5 i5 e Answer received: !yro11 Command to send: c o11 _1 e Answer received: !ysspark.submit.pyFiles Command to send: c o11 _2 e Answer received: !ys Command to send: a e o5 e Answer received: !yi8 Command to send: a g o5 i6 e Answer received: !yro12 Command to send: c o12 _1 e Answer received: !ysspark.submit.deployMode Command to send: c o12 _2 e Answer received: !ysclient Command to send: a e o5 e Answer received: !yi8 Command to send: a g o5 i7 e Answer received: !yro13 Command to send: c o13 _1 e Answer received: !ysspark.ui.showConsoleProgress Command to send: c o13 _2 e Answer received: !ystrue Command to send: a e o5 e Answer received: !yi8 Command to send: r u JavaSparkContext rj e Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Answer received: !ycorg.apache.spark.api.java.JavaSparkContext Command to send: i org.apache.spark.api.java.JavaSparkContext ro0 e run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query REVOKE SELECT(x2) ON tbl FROM R2, R3, R1 on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_3' and table = 'table_for_recompression' on node2 Stdout:763 Executing query DROP ROLE R1 on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query GRANT SELECT(x2) ON tbl TO R2 on instance Executing query CREATE ROLE R1 on instance Executing query GRANT R1 TO A on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_3' and table = 'table_for_recompression' on node2 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query DROP ROLE R3 on instance Executing query CREATE ROLE R3 on instance Command to send: A dfbef6cc0ef2972b9e9510f4805662b425a472689b5e93b01940c781834f8340 Answer received: !yv Command to send: m d o1 e Answer received: !yv Command to send: m d o2 e Answer received: !yv Command to send: m d o3 e Answer received: !yv Command to send: m d o4 e Answer received: !yv Command to send: m d o6 e Answer received: !yv Command to send: m d o7 e Answer received: !yv Command to send: m d o8 e Answer received: !yv Command to send: m d o5 e Answer received: !yv Executing query GRANT R3 TO R2 on instance Connection dropped: socket connection error: No route to host Connection dropped: socket connection error: No route to host Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Stdout:763 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_3' and table = 'table_for_recompression' on node2 Executing query GRANT SELECT(x3) ON tbl TO R1 on instance Executing query SELECT default_compression_codec FROM system.parts where name = 'all_1_1_3' and table = 'table_for_recompression' on node2 Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Executing query REVOKE SELECT(x2) ON tbl FROM R2, R1, R3 on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Answer received: !yro14 Command to send: c o14 sc e Answer received: !yro15 Command to send: c o15 conf e Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 Answer received: !yro16 Command to send: r u PythonAccumulatorV2 rj e Answer received: !ycorg.apache.spark.api.python.PythonAccumulatorV2 Command to send: i org.apache.spark.api.python.PythonAccumulatorV2 s127.0.0.1 i58295 sdfbef6cc0ef2972b9e9510f4805662b425a472689b5e93b01940c781834f8340 e Answer received: !yro17 Command to send: c o14 sc e Answer received: !yro18 Command to send: c o18 register ro17 e Answer received: !yv Command to send: r u PythonUtils rj e Answer received: !ycorg.apache.spark.api.python.PythonUtils Command to send: r m org.apache.spark.api.python.PythonUtils isEncryptionEnabled e Answer received: !ym Command to send: c z:org.apache.spark.api.python.PythonUtils isEncryptionEnabled ro14 e Answer received: !ybfalse Command to send: r u PythonUtils rj e Answer received: !ycorg.apache.spark.api.python.PythonUtils Command to send: r m org.apache.spark.api.python.PythonUtils getPythonAuthSocketTimeout e Answer received: !ym Command to send: c z:org.apache.spark.api.python.PythonUtils getPythonAuthSocketTimeout ro14 e Answer received: !yL15 Command to send: r u PythonUtils rj e Answer received: !ycorg.apache.spark.api.python.PythonUtils Command to send: r m org.apache.spark.api.python.PythonUtils getSparkBufferSize e Answer received: !ym Command to send: c z:org.apache.spark.api.python.PythonUtils getSparkBufferSize ro14 e Answer received: !yi65536 Command to send: r u org rj e Answer received: !yp Command to send: r u org.apache rj e Answer received: !yp Command to send: r u org.apache.spark rj e Answer received: !yp Command to send: r u org.apache.spark.SparkFiles rj e Answer received: !ycorg.apache.spark.SparkFiles Command to send: r m org.apache.spark.SparkFiles getRootDirectory e Answer received: !ym Command to send: c z:org.apache.spark.SparkFiles getRootDirectory e Answer received: !ys/tmp/spark-c9a9acd3-6eae-450f-907b-3fbd9af88b28/userFiles-bbb8fe24-89c1-43c8-82c5-35cb07f70746 Command to send: c o16 get sspark.submit.pyFiles s e Answer received: !ys Command to send: r u org rj e Answer received: !yp Command to send: r u org.apache rj e Answer received: !yp Command to send: r u org.apache.spark rj e Answer received: !yp Command to send: r u org.apache.spark.util rj e Answer received: !yp Command to send: r u org.apache.spark.util.Utils rj e Answer received: !ycorg.apache.spark.util.Utils Command to send: r m org.apache.spark.util.Utils getLocalDir e Answer received: !ym Command to send: c o14 sc e Answer received: !yro19 Command to send: c o19 conf e Answer received: !yro20 Command to send: c z:org.apache.spark.util.Utils getLocalDir ro20 e Answer received: !ys/tmp/spark-c9a9acd3-6eae-450f-907b-3fbd9af88b28 Command to send: r u org rj e Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Answer received: !yp Command to send: r u org.apache rj e Answer received: !yp Command to send: r u org.apache.spark rj e Answer received: !yp Command to send: r u org.apache.spark.util rj e Answer received: !yp Command to send: r u org.apache.spark.util.Utils rj e Answer received: !ycorg.apache.spark.util.Utils Command to send: r m org.apache.spark.util.Utils createTempDir e Answer received: !ym Command to send: c z:org.apache.spark.util.Utils createTempDir s/tmp/spark-c9a9acd3-6eae-450f-907b-3fbd9af88b28 spyspark e Answer received: !yro21 Command to send: c o21 getAbsolutePath e Answer received: !ys/tmp/spark-c9a9acd3-6eae-450f-907b-3fbd9af88b28/pyspark-a10d9318-6b5a-48b3-b0da-f89424abd8a0 Command to send: c o16 get sspark.python.profile sfalse e Answer received: !ysfalse Command to send: r u SparkSession rj e Answer received: !ycorg.apache.spark.sql.SparkSession Command to send: r m org.apache.spark.sql.SparkSession getDefaultSession e Answer received: !ym Command to send: c z:org.apache.spark.sql.SparkSession getDefaultSession e Answer received: !yro22 Command to send: c o22 isDefined e Answer received: !ybfalse Command to send: r u SparkSession rj e Answer received: !ycorg.apache.spark.sql.SparkSession Command to send: c o14 sc e Answer received: !yro23 Command to send: i java.util.HashMap e Answer received: !yao24 Command to send: c o24 put sspark.app.name sspark_test e Answer received: !yn Command to send: c o24 put sspark.master slocal e Answer received: !yn Command to send: i org.apache.spark.sql.SparkSession ro23 ro24 e Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Answer received: !yro25 Command to send: r u SparkSession rj e Answer received: !ycorg.apache.spark.sql.SparkSession Command to send: r m org.apache.spark.sql.SparkSession setDefaultSession e Answer received: !ym Command to send: c z:org.apache.spark.sql.SparkSession setDefaultSession ro25 e Answer received: !yv Command to send: r u SparkSession rj e Answer received: !ycorg.apache.spark.sql.SparkSession Command to send: r m org.apache.spark.sql.SparkSession setActiveSession e Answer received: !ym Command to send: c z:org.apache.spark.sql.SparkSession setActiveSession ro25 e Answer received: !yv Command to send: c o14 stop e Executing query GRANT SELECT(x9) ON tbl TO R1 on instance Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Answer received: !yv Executing query GRANT R3 TO C on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query GRANT SELECT(x2) ON tbl TO R3 on instance Command to send: r u SparkSession rj e Answer received: !ycorg.apache.spark.sql.SparkSession Command to send: r m org.apache.spark.sql.SparkSession clearDefaultSession e Answer received: !ym Command to send: c z:org.apache.spark.sql.SparkSession clearDefaultSession e Answer received: !yv Command to send: r u SparkSession rj e Answer received: !ycorg.apache.spark.sql.SparkSession Command to send: r m org.apache.spark.sql.SparkSession clearActiveSession e Answer received: !ym Command to send: c z:org.apache.spark.sql.SparkSession clearActiveSession e Answer received: !yv clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Cluster name: project_name:roottests3accessheaders-gw4. Added instance name:node1 tag:8b2301119731 base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/.env', '--project-name', 'roottests3accessheaders-gw4', '--file', '/ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ Starting cluster... Running tests in /ClickHouse/tests/integration/test_s3_access_headers/test.py Cluster start called. is_up=False Docker networks for project roottests3accessheaders-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottests3accessheaders-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query REVOKE SELECT(x5) ON tbl FROM R2, R1, R3 on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 Command to send: m d o24 e Answer received: !yv Docker volumes for project roottests3accessheaders-gw4 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottests3accessheaders-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottests3accessheaders-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Docker volumes for project roottests3accessheaders-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottests3accessheaders-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottests3accessheaders-gw4 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Stdout:3 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 3 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_s3_access_headers/configs/config.d/named_collections.xml', '/ClickHouse/tests/integration/test_s3_access_headers/configs/config.d/s3_headers.xml'] to /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/node1/database Setup logs dir /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/minio/certs/public.crt', 'RESOLVER_LOGS': '/ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/resolver', 'RESOLVER_LOGS_FS': 'bind'} stored in /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/.env --project-name roottests3accessheaders-gw4 --file /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml pull] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/exec/b5d757ce4d9b05907a48d5573bd28047f239e8a0400ee6e7a25d818c118f0278/json HTTP/1.1" 200 584 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query GRANT SELECT(x1) ON tbl TO R2 on instance No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrefreshablemv-gw5-node1-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/c5cf8ae238c3a8c70792b442d4d10aece49f23632167c1c825816c6755c92887/start HTTP/1.1" 200 0 Executing query GRANT SELECT(x5) ON tbl TO R1 on instance http://localhost:None "GET /v1.46/exec/c5cf8ae238c3a8c70792b442d4d10aece49f23632167c1c825816c6755c92887/json HTTP/1.1" 200 586 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 Stderr: Container roottestrestorereplica-gw9-replica1-1 Stopping Stderr: Container roottestrestorereplica-gw9-replica3-1 Stopping Stderr: Container roottestrestorereplica-gw9-replica2-1 Stopping Stderr: Container roottestrestorereplica-gw9-replica3-1 Stopped Stderr: Container roottestrestorereplica-gw9-replica2-1 Stopped Stderr: Container roottestrestorereplica-gw9-replica1-1 Stopped Stderr: Container roottestrestorereplica-gw9-zoo2-1 Stopping Stderr: Container roottestrestorereplica-gw9-zoo3-1 Stopping Stderr: Container roottestrestorereplica-gw9-zoo1-1 Stopping Stderr: Container roottestrestorereplica-gw9-zoo3-1 Stopped Stderr: Container roottestrestorereplica-gw9-zoo2-1 Stopped Stderr: Container roottestrestorereplica-gw9-zoo1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/.env --project-name roottestrestorereplica-gw9 --file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica2/docker-compose.yml --file /ClickHouse/tests/integration/test_restore_replica/_instances-0-gw9/replica3/docker-compose.yml down --volumes] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query REVOKE SELECT(x6) ON tbl FROM R2, R1, R3 on instance Executing query REVOKE SELECT(x2) ON tbl FROM R2, R1, R3 on instance Command to send: m d o10 e Answer received: !yv Command to send: m d o11 e Answer received: !yv Command to send: m d o12 e Answer received: !yv Command to send: m d o13 e Answer received: !yv Command to send: m d o15 e Answer received: !yv Command to send: m d o18 e Answer received: !yv Command to send: m d o19 e Answer received: !yv Command to send: m d o20 e Answer received: !yv Command to send: m d o21 e Answer received: !yv Command to send: m d o22 e Answer received: !yv Command to send: m d o23 e Answer received: !yv Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Stderr: Container roottestrestorereplica-gw9-replica3-1 Stopping Stderr: Container roottestrestorereplica-gw9-replica2-1 Stopping Stderr: Container roottestrestorereplica-gw9-replica1-1 Stopping Stderr: Container roottestrestorereplica-gw9-replica1-1 Stopped Stderr: Container roottestrestorereplica-gw9-replica1-1 Removing Stderr: Container roottestrestorereplica-gw9-replica2-1 Stopped Stderr: Container roottestrestorereplica-gw9-replica2-1 Removing Stderr: Container roottestrestorereplica-gw9-replica3-1 Stopped Stderr: Container roottestrestorereplica-gw9-replica3-1 Removing Stderr: Container roottestrestorereplica-gw9-replica2-1 Removed Stderr: Container roottestrestorereplica-gw9-replica3-1 Removed Stderr: Container roottestrestorereplica-gw9-replica1-1 Removed Stderr: Container roottestrestorereplica-gw9-zoo2-1 Stopping Stderr: Container roottestrestorereplica-gw9-zoo3-1 Stopping Stderr: Container roottestrestorereplica-gw9-zoo1-1 Stopping Stderr: Container roottestrestorereplica-gw9-zoo3-1 Stopped Stderr: Container roottestrestorereplica-gw9-zoo3-1 Removing Stderr: Container roottestrestorereplica-gw9-zoo1-1 Stopped Stderr: Container roottestrestorereplica-gw9-zoo1-1 Removing Stderr: Container roottestrestorereplica-gw9-zoo2-1 Stopped Stderr: Container roottestrestorereplica-gw9-zoo2-1 Removing Stderr: Container roottestrestorereplica-gw9-zoo3-1 Removed Stderr: Container roottestrestorereplica-gw9-zoo2-1 Removed Stderr: Container roottestrestorereplica-gw9-zoo1-1 Removed Stderr: Network roottestrestorereplica-gw9_default Removing Stderr: Network roottestrestorereplica-gw9_default Removed Cleanup called Docker networks for project roottestrestorereplica-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrestorereplica-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Docker volumes for project roottestrestorereplica-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrestorereplica-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Unstopped containers: {} No running containers for project: roottestrestorereplica-gw9 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1631 Clickhouse process running. run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:3 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 3 test_restart_server/test.py::test_drop_memory_database Running tests in /ClickHouse/tests/integration/test_restart_server/test.py Cluster start called. is_up=False Executing query GRANT SELECT(x6) ON tbl TO R3 on instance Stdout:1631 Executing query select 20 on node1 Docker networks for project roottestrestartserver-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrestartserver-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrestartserver-gw9 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestrestartserver-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrestartserver-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query GRANT SELECT(x10) ON tbl TO R2 on instance run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c ps -C clickhouse] Docker volumes for project roottestrestartserver-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrestartserver-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestrestartserver-gw9 Trying to prune unused networks... Stdout: PID TTY TIME CMD Stdout: 1631 ? 00:00:00 clickhouse run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c pkill clickhouse] Trying to prune unused images... Command:[docker image prune -f] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:3 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 3 Setup directory for instance: node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/node/database Setup logs dir /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/node/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/.env --project-name roottestrestartserver-gw9 --file /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/node/docker-compose.yml pull] Stdout:1631 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query GRANT SELECT(x8) ON tbl TO R1 on instance Executing query GRANT SELECT(x2) ON tbl TO R2 on instance Stderr: node2 Skipped - Image is already being pulled by node1 Stderr: proxy1 Skipped - Image is already being pulled by proxy2 Stderr: proxy2 Skipped - Image is already being pulled by proxy1 Stderr: resolver Pulling Stderr: proxy1 Pulling Stderr: minio1 Pulling Stderr: node1 Pulling Stderr: proxy1 Pulled Stderr: minio1 Pulled Stderr: resolver Pulled Stderr: node1 Pulled Stderr: node Pulling Trying to create Minio instance by command docker compose --project-name roottests3accessheaders-gw4 --env-file /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d Stderr: node Pulled Command:[docker compose --project-name roottests3accessheaders-gw4 --env-file /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d] ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/.env --project-name roottestprometheusendpoint-gw2 --file /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/.env --project-name roottestprometheusendpoint-gw2 --file /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/node/docker-compose.yml up -d --no-recreate] Stderr: zoo3 Skipped - Image is already being pulled by node1 Stderr: zoo1 Skipped - Image is already being pulled by node1 Stderr: zoo2 Skipped - Image is already being pulled by node1 Stderr: resolver Pulling Stderr: proxy2 Pulling Stderr: minio1 Pulling Stderr: node1 Pulling Stderr: proxy2 Pulled Stderr: node1 Pulled Stderr: minio1 Pulled Stderr: resolver Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper1/log', '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper1/config', '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper1/coordination', '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper2/log', '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper2/config', '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper2/coordination', '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper3/log', '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper3/config', '/ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/keeper3/coordination'] Command:[docker compose --project-name roottestreplicatedzerocopyprojectionmutation-gw8 --env-file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr: node Pulling Stderr: node Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/.env --project-name roottestrocksdbreadonly-gw3 --file /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/.env --project-name roottestrocksdbreadonly-gw3 --file /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/node/docker-compose.yml up -d --no-recreate] Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 Stderr: Network roottestprometheusendpoint-gw2_default Creating Stderr: Network roottestprometheusendpoint-gw2_default Created Stderr: Container roottestprometheusendpoint-gw2-node-1 Creating Stderr: Container roottestprometheusendpoint-gw2-node-1 Created Stderr: Container roottestprometheusendpoint-gw2-node-1 Starting Stderr: Container roottestprometheusendpoint-gw2-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestprometheusendpoint-gw2-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestprometheusendpoint-gw2-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.3.2... http://localhost:None "GET /v1.46/containers/roottestprometheusendpoint-gw2-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b576dcc431d382429bb09e55e57c6d069c6d07f1ebe63ad08346e396097745cc/json HTTP/1.1" 200 None Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node http://localhost:None "GET /v1.46/containers/b576dcc431d382429bb09e55e57c6d069c6d07f1ebe63ad08346e396097745cc/json HTTP/1.1" 200 None run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False http://localhost:None "GET /v1.46/containers/b576dcc431d382429bb09e55e57c6d069c6d07f1ebe63ad08346e396097745cc/json HTTP/1.1" 200 None Stdout:1631 Stderr: Network roottestrocksdbreadonly-gw3_default Creating Stderr: Network roottestrocksdbreadonly-gw3_default Created Stderr: Container roottestrocksdbreadonly-gw3-node-1 Creating Stderr: Container roottestrocksdbreadonly-gw3-node-1 Created Stderr: Container roottestrocksdbreadonly-gw3-node-1 Starting Stderr: Container roottestrocksdbreadonly-gw3-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestrocksdbreadonly-gw3-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestrocksdbreadonly-gw3-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.4.2... http://localhost:None "GET /v1.46/containers/roottestrocksdbreadonly-gw3-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0f52acd97435411111d59640f45972d161485c3f2b13606d23fb88ab168b2571/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b576dcc431d382429bb09e55e57c6d069c6d07f1ebe63ad08346e396097745cc/json HTTP/1.1" 200 None Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 http://localhost:None "GET /v1.46/containers/0f52acd97435411111d59640f45972d161485c3f2b13606d23fb88ab168b2571/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b576dcc431d382429bb09e55e57c6d069c6d07f1ebe63ad08346e396097745cc/json HTTP/1.1" 200 None ClickHouse node started Starting new HTTP connection (1): 172.16.3.2:8001 http://localhost:None "GET /v1.46/containers/0f52acd97435411111d59640f45972d161485c3f2b13606d23fb88ab168b2571/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0f52acd97435411111d59640f45972d161485c3f2b13606d23fb88ab168b2571/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0f52acd97435411111d59640f45972d161485c3f2b13606d23fb88ab168b2571/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0f52acd97435411111d59640f45972d161485c3f2b13606d23fb88ab168b2571/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/0f52acd97435411111d59640f45972d161485c3f2b13606d23fb88ab168b2571/json HTTP/1.1" 200 None ClickHouse node started Executing query CREATE TABLE test (key UInt64, value String) Engine=EmbeddedRocksDB(0, '/var/lib/clickhouse/store/test_rocksdb_read_only_missing') PRIMARY KEY(key); on node Starting new HTTP connection (1): 172.16.3.2:8001 http://172.16.3.2:8001 "GET /metrics HTTP/1.1" 200 None Executing query SELECT 1 on node Stderr:time="2025-04-02T03:58:58Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestreplicatedzerocopyprojectionmutation-gw8_default Creating Stderr: Network roottestreplicatedzerocopyprojectionmutation-gw8_default Created Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo3-1 Creating Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo1-1 Creating Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo2-1 Creating Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo2-1 Created Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo3-1 Created Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo1-1 Created Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo3-1 Starting Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo1-1 Starting Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo2-1 Starting Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo2-1 Started Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo1-1 Started Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo3-1 Started Stderr:time="2025-04-02T03:59:00Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:59:00Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestreplicatedzerocopyprojectionmutation-gw8-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.5.2, port:2181, use_ssl:False Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SELECT 2 on node Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrocksdbreadonly-gw3-node-1 bash -c ps -C clickhouse] Connection dropped: socket connection error: Connection refused Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Stdout: PID TTY TIME CMD Stdout: 9 ? 00:00:00 clickhouse run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrocksdbreadonly-gw3-node-1 bash -c pkill clickhouse] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT 3 on node Stdout:1631 Stdout:9 Starting new HTTP connection (1): 172.16.3.2:8001 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance http://172.16.3.2:8001 "GET /metrics HTTP/1.1" 200 None Executing query SELECT throwIf(1, 'test', toInt16(42)) SETTINGS allow_custom_error_code_in_throwif=1 on node Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Starting new HTTP connection (1): 172.16.3.2:8001 http://172.16.3.2:8001 "GET /metrics HTTP/1.1" 200 None [gw2] PASSED test_prometheus_endpoint/test.py::test_prometheus_endpoint Command:[docker compose --env-file /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/.env --project-name roottestprometheusendpoint-gw2 --file /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/node/docker-compose.yml stop --timeout 20] Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 Stderr:time="2025-04-02T03:58:58Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottests3accessheaders-gw4_default Creating Stderr: Network roottests3accessheaders-gw4_default Created Stderr: Volume "roottests3accessheaders-gw4_data1-1" Creating Stderr: Volume "roottests3accessheaders-gw4_data1-1" Created Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Stderr: Container roottests3accessheaders-gw4-proxy1-1 Creating Stderr: Container roottests3accessheaders-gw4-proxy2-1 Creating Stderr: Container roottests3accessheaders-gw4-proxy1-1 Created Stderr: Container roottests3accessheaders-gw4-proxy2-1 Created Stderr: Container roottests3accessheaders-gw4-minio1-1 Creating Stderr: Container roottests3accessheaders-gw4-resolver-1 Creating Stderr: Container roottests3accessheaders-gw4-resolver-1 Created Stderr: Container roottests3accessheaders-gw4-minio1-1 Created Stderr: Container roottests3accessheaders-gw4-proxy1-1 Starting Stderr: Container roottests3accessheaders-gw4-proxy2-1 Starting Stderr: Container roottests3accessheaders-gw4-proxy2-1 Started Stderr: Container roottests3accessheaders-gw4-proxy1-1 Started Stderr: Container roottests3accessheaders-gw4-minio1-1 Starting Stderr: Container roottests3accessheaders-gw4-resolver-1 Starting Stderr: Container roottests3accessheaders-gw4-resolver-1 Started Stderr: Container roottests3accessheaders-gw4-minio1-1 Started Stderr:time="2025-04-02T03:59:01Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:59:01Z" level=debug msg="otel error" error="" Trying to connect to Minio... get_instance_ip instance_name=minio1 http://localhost:None "GET /v1.46/containers/roottests3accessheaders-gw4-minio1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=proxy1 http://localhost:None "GET /v1.46/containers/roottests3accessheaders-gw4-proxy1-1/json HTTP/1.1" 200 None Starting new HTTP connection (1): 172.16.7.5:9001 Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (2): 172.16.7.5:9001 Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (3): 172.16.7.5:9001 Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (4): 172.16.7.5:9001 Can't connect to Minio: HTTPConnectionPool(host='172.16.7.5', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) Executing query DROP ROLE R3 on instance Executing query CREATE ROLE R3 on instance Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query GRANT R3 TO R2 on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Stdout:1631 Stdout:9 Executing query GRANT SELECT(x6) ON tbl TO R2 on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 Executing query DROP ROLE R3 on instance Starting new HTTP connection (5): 172.16.7.5:9001 http://172.16.7.5:9001 "GET / HTTP/1.1" 200 0 Connected to Minio. http://172.16.7.5:9001 "GET /root?location= HTTP/1.1" 404 0 http://172.16.7.5:9001 "PUT /root HTTP/1.1" 200 0 S3 bucket 'root' created http://172.16.7.5:9001 "GET /root2?location= HTTP/1.1" 404 0 http://172.16.7.5:9001 "PUT /root2 HTTP/1.1" 200 0 S3 bucket 'root2' created ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/.env --project-name roottests3accessheaders-gw4 --file /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/.env --project-name roottests3accessheaders-gw4 --file /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml up -d --no-recreate] Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 Stderr: Container roottests3accessheaders-gw4-proxy2-1 Running Stderr: Container roottests3accessheaders-gw4-proxy1-1 Running Stderr: Container roottests3accessheaders-gw4-minio1-1 Running Stderr: Container roottests3accessheaders-gw4-resolver-1 Running Stderr: Container roottests3accessheaders-gw4-node1-1 Creating Stderr: Container roottests3accessheaders-gw4-node1-1 Created Stderr: Container roottests3accessheaders-gw4-node1-1 Starting Stderr: Container roottests3accessheaders-gw4-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottests3accessheaders-gw4-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottests3accessheaders-gw4-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.7.6... http://localhost:None "GET /v1.46/containers/roottests3accessheaders-gw4-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ae96b332b6058dd3ac77871b7754404970f249efcd715c94b7dbf527c921426e/json HTTP/1.1" 200 None run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/containers/ae96b332b6058dd3ac77871b7754404970f249efcd715c94b7dbf527c921426e/json HTTP/1.1" 200 None run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:1631 run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'rm -r /var/lib/clickhouse/store/test_rocksdb_read_only_missing'] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c rm -r /var/lib/clickhouse/store/test_rocksdb_read_only_missing] http://localhost:None "GET /v1.46/containers/ae96b332b6058dd3ac77871b7754404970f249efcd715c94b7dbf527c921426e/json HTTP/1.1" 200 None run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrocksdbreadonly-gw3-node-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/8c48718f5e3ba920ec1cab6e4e11d60231841a2c4df0cc590c4eafca01af2bd9/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/8c48718f5e3ba920ec1cab6e4e11d60231841a2c4df0cc590c4eafca01af2bd9/json HTTP/1.1" 200 586 http://localhost:None "GET /v1.46/containers/ae96b332b6058dd3ac77871b7754404970f249efcd715c94b7dbf527c921426e/json HTTP/1.1" 200 None Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 http://localhost:None "GET /v1.46/containers/ae96b332b6058dd3ac77871b7754404970f249efcd715c94b7dbf527c921426e/json HTTP/1.1" 200 None ClickHouse node1 started http://172.16.7.5:9001 "PUT /root?policy= HTTP/1.1" 204 0 http://172.16.7.5:9001 "GET /root-with-auth?location= HTTP/1.1" 404 0 http://172.16.7.5:9001 "PUT /root-with-auth HTTP/1.1" 200 0 S3 bucket created Starting mock server mocker_s3.py run container_id:roottests3accessheaders-gw4-resolver-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname mocker_s3.py) && echo aW1wb3J0IGh0dHAuY2xpZW50CmltcG9ydCBodHRwLnNlcnZlcgppbXBvcnQgcmFuZG9tCmltcG9ydCBzb2NrZXRzZXJ2ZXIKaW1wb3J0IHN5cwppbXBvcnQgdXJsbGliLnBhcnNlCgpVUFNUUkVBTV9IT1NUID0gIm1pbmlvMTo5MDAxIgpyYW5kb20uc2VlZCgiTm8gbGlzdCBvYmplY3RzLzEuMCIpCgoKZGVmIHJlcXVlc3QoY29tbWFuZCwgdXJsLCBoZWFkZXJzPXt9LCBkYXRhPU5vbmUpOgogICAgIiIiTWluaS1yZXF1ZXN0cy4iIiIKCiAgICBjbGFzcyBEdW1teToKICAgICAgICBwYXNzCgogICAgcGFydHMgPSB1cmxsaWIucGFyc2UudXJscGFyc2UodXJsKQogICAgYyA9IGh0dHAuY2xpZW50LkhUVFBDb25uZWN0aW9uKHBhcnRzLmhvc3RuYW1lLCBwYXJ0cy5wb3J0KQogICAgYy5yZXF1ZXN0KAogICAgICAgIGNvbW1hbmQsCiAgICAgICAgdXJsbGliLnBhcnNlLnVybHVucGFyc2UocGFydHMuX3JlcGxhY2Uoc2NoZW1lPSIiLCBuZXRsb2M9IiIpKSwKICAgICAgICBoZWFkZXJzPWhlYWRlcnMsCiAgICAgICAgYm9keT1kYXRhLAogICAgKQogICAgciA9IGMuZ2V0cmVzcG9uc2UoKQogICAgcmVzdWx0ID0gRHVtbXkoKQogICAgcmVzdWx0LnN0YXR1c19jb2RlID0gci5zdGF0dXMKICAgIHJlc3VsdC5oZWFkZXJzID0gci5oZWFkZXJzCiAgICByZXN1bHQuY29udGVudCA9IHIucmVhZCgpCiAgICByZXR1cm4gcmVzdWx0CgoKQ1VTVE9NX0FVVEhfVE9LRU5fSEVBREVSID0gImN1c3RvbS1hdXRoLXRva2VuIgpDVVNUT01fQVVUSF9UT0tFTl9WQUxJRF9WQUxVRSA9ICJWYWxpZFRva2VuMTIzNCIKCgpjbGFzcyBSZXF1ZXN0SGFuZGxlcihodHRwLnNlcnZlci5CYXNlSFRUUFJlcXVlc3RIYW5kbGVyKToKICAgIGRlZiBkb19HRVQoc2VsZik6CiAgICAgICAgaWYgc2VsZi5wYXRoID09ICIvIjoKICAgICAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDIwMCkKICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQvcGxhaW4iKQogICAgICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKICAgICAgICAgICAgc2VsZi53ZmlsZS53cml0ZShiIk9LIikKICAgICAgICAgICAgcmV0dXJuCiAgICAgICAgc2VsZi5kb19IRUFEKCkKCiAgICBkZWYgZG9fUFVUKHNlbGYpOgogICAgICAgIHNlbGYuZG9fSEVBRCgpCgogICAgZGVmIGRvX0RFTEVURShzZWxmKToKICAgICAgICBzZWxmLmRvX0hFQUQoKQoKICAgIGRlZiBkb19QT1NUKHNlbGYpOgogICAgICAgIHNlbGYuZG9fSEVBRCgpCgogICAgZGVmIGRvX0hFQUQoc2VsZik6CgogICAgICAgIGN1c3RvbV9hdXRoX3Rva2VuID0gc2VsZi5oZWFkZXJzLmdldChDVVNUT01fQVVUSF9UT0tFTl9IRUFERVIpCiAgICAgICAgaWYgY3VzdG9tX2F1dGhfdG9rZW4gYW5kIGN1c3RvbV9hdXRoX3Rva2VuICE9IENVU1RPTV9BVVRIX1RPS0VOX1ZBTElEX1ZBTFVFOgogICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoNDAzKQogICAgICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LVR5cGUiLCAiYXBwbGljYXRpb24veG1sIikKICAgICAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgICAgICAgICBib2R5ID0gZiIiIjw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+CjxFcnJvcj4KICAgIDxDb2RlPkFjY2Vzc0RlbmllZDwvQ29kZT4KICAgIDxNZXNzYWdlPkFjY2VzcyBEZW5pZWQuIEN1c3RvbSB0b2tlbiB3YXMge2N1c3RvbV9hdXRoX3Rva2VufSwgdGhlIGNvcnJlY3Qgb25lOiB7Q1VTVE9NX0FVVEhfVE9LRU5fVkFMSURfVkFMVUV9LjwvTWVzc2FnZT4KICAgIDxSZXNvdXJjZT5SRVNPVVJDRTwvUmVzb3VyY2U+CiAgICA8UmVxdWVzdElkPlJFUVVFU1RfSUQ8L1JlcXVlc3RJZD4KPC9FcnJvcj4KIiIiCiAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYm9keS5lbmNvZGUoKSkKICAgICAgICAgICAgcmV0dXJuCgogICAgICAgIGNvbnRlbnRfbGVuZ3RoID0gc2VsZi5oZWFkZXJzLmdldCgiQ29udGVudC1MZW5ndGgiKQogICAgICAgIGRhdGEgPSBzZWxmLnJmaWxlLnJlYWQoaW50KGNvbnRlbnRfbGVuZ3RoKSkgaWYgY29udGVudF9sZW5ndGggZWxzZSBOb25lCiAgICAgICAgciA9IHJlcXVlc3QoCiAgICAgICAgICAgIHNlbGYuY29tbWFuZCwKICAgICAgICAgICAgZiJodHRwOi8ve1VQU1RSRUFNX0hPU1R9e3NlbGYucGF0aH0iLAogICAgICAgICAgICBoZWFkZXJzPXNlbGYuaGVhZGVycywKICAgICAgICAgICAgZGF0YT1kYXRhLAogICAgICAgICkKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2Uoci5zdGF0dXNfY29kZSkKICAgICAgICBmb3IgaywgdiBpbiByLmhlYWRlcnMuaXRlbXMoKToKICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcihrLCB2KQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoci5jb250ZW50KQogICAgICAgIHNlbGYud2ZpbGUuY2xvc2UoKQoKCmNsYXNzIFRocmVhZGVkSFRUUFNlcnZlcihzb2NrZXRzZXJ2ZXIuVGhyZWFkaW5nTWl4SW4sIGh0dHAuc2VydmVyLkhUVFBTZXJ2ZXIpOgogICAgIiIiSGFuZGxlIHJlcXVlc3RzIGluIGEgc2VwYXJhdGUgdGhyZWFkLiIiIgoKCmh0dHBkID0gVGhyZWFkZWRIVFRQU2VydmVyKCgiMC4wLjAuMCIsIGludChzeXMuYXJndlsxXSkpLCBSZXF1ZXN0SGFuZGxlcikKaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > mocker_s3.py'] Command:[docker exec roottests3accessheaders-gw4-resolver-1 bash -c mkdir -p $(dirname mocker_s3.py) && echo aW1wb3J0IGh0dHAuY2xpZW50CmltcG9ydCBodHRwLnNlcnZlcgppbXBvcnQgcmFuZG9tCmltcG9ydCBzb2NrZXRzZXJ2ZXIKaW1wb3J0IHN5cwppbXBvcnQgdXJsbGliLnBhcnNlCgpVUFNUUkVBTV9IT1NUID0gIm1pbmlvMTo5MDAxIgpyYW5kb20uc2VlZCgiTm8gbGlzdCBvYmplY3RzLzEuMCIpCgoKZGVmIHJlcXVlc3QoY29tbWFuZCwgdXJsLCBoZWFkZXJzPXt9LCBkYXRhPU5vbmUpOgogICAgIiIiTWluaS1yZXF1ZXN0cy4iIiIKCiAgICBjbGFzcyBEdW1teToKICAgICAgICBwYXNzCgogICAgcGFydHMgPSB1cmxsaWIucGFyc2UudXJscGFyc2UodXJsKQogICAgYyA9IGh0dHAuY2xpZW50LkhUVFBDb25uZWN0aW9uKHBhcnRzLmhvc3RuYW1lLCBwYXJ0cy5wb3J0KQogICAgYy5yZXF1ZXN0KAogICAgICAgIGNvbW1hbmQsCiAgICAgICAgdXJsbGliLnBhcnNlLnVybHVucGFyc2UocGFydHMuX3JlcGxhY2Uoc2NoZW1lPSIiLCBuZXRsb2M9IiIpKSwKICAgICAgICBoZWFkZXJzPWhlYWRlcnMsCiAgICAgICAgYm9keT1kYXRhLAogICAgKQogICAgciA9IGMuZ2V0cmVzcG9uc2UoKQogICAgcmVzdWx0ID0gRHVtbXkoKQogICAgcmVzdWx0LnN0YXR1c19jb2RlID0gci5zdGF0dXMKICAgIHJlc3VsdC5oZWFkZXJzID0gci5oZWFkZXJzCiAgICByZXN1bHQuY29udGVudCA9IHIucmVhZCgpCiAgICByZXR1cm4gcmVzdWx0CgoKQ1VTVE9NX0FVVEhfVE9LRU5fSEVBREVSID0gImN1c3RvbS1hdXRoLXRva2VuIgpDVVNUT01fQVVUSF9UT0tFTl9WQUxJRF9WQUxVRSA9ICJWYWxpZFRva2VuMTIzNCIKCgpjbGFzcyBSZXF1ZXN0SGFuZGxlcihodHRwLnNlcnZlci5CYXNlSFRUUFJlcXVlc3RIYW5kbGVyKToKICAgIGRlZiBkb19HRVQoc2VsZik6CiAgICAgICAgaWYgc2VsZi5wYXRoID09ICIvIjoKICAgICAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDIwMCkKICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQvcGxhaW4iKQogICAgICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKICAgICAgICAgICAgc2VsZi53ZmlsZS53cml0ZShiIk9LIikKICAgICAgICAgICAgcmV0dXJuCiAgICAgICAgc2VsZi5kb19IRUFEKCkKCiAgICBkZWYgZG9fUFVUKHNlbGYpOgogICAgICAgIHNlbGYuZG9fSEVBRCgpCgogICAgZGVmIGRvX0RFTEVURShzZWxmKToKICAgICAgICBzZWxmLmRvX0hFQUQoKQoKICAgIGRlZiBkb19QT1NUKHNlbGYpOgogICAgICAgIHNlbGYuZG9fSEVBRCgpCgogICAgZGVmIGRvX0hFQUQoc2VsZik6CgogICAgICAgIGN1c3RvbV9hdXRoX3Rva2VuID0gc2VsZi5oZWFkZXJzLmdldChDVVNUT01fQVVUSF9UT0tFTl9IRUFERVIpCiAgICAgICAgaWYgY3VzdG9tX2F1dGhfdG9rZW4gYW5kIGN1c3RvbV9hdXRoX3Rva2VuICE9IENVU1RPTV9BVVRIX1RPS0VOX1ZBTElEX1ZBTFVFOgogICAgICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2UoNDAzKQogICAgICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LVR5cGUiLCAiYXBwbGljYXRpb24veG1sIikKICAgICAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgICAgICAgICBib2R5ID0gZiIiIjw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+CjxFcnJvcj4KICAgIDxDb2RlPkFjY2Vzc0RlbmllZDwvQ29kZT4KICAgIDxNZXNzYWdlPkFjY2VzcyBEZW5pZWQuIEN1c3RvbSB0b2tlbiB3YXMge2N1c3RvbV9hdXRoX3Rva2VufSwgdGhlIGNvcnJlY3Qgb25lOiB7Q1VTVE9NX0FVVEhfVE9LRU5fVkFMSURfVkFMVUV9LjwvTWVzc2FnZT4KICAgIDxSZXNvdXJjZT5SRVNPVVJDRTwvUmVzb3VyY2U+CiAgICA8UmVxdWVzdElkPlJFUVVFU1RfSUQ8L1JlcXVlc3RJZD4KPC9FcnJvcj4KIiIiCiAgICAgICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYm9keS5lbmNvZGUoKSkKICAgICAgICAgICAgcmV0dXJuCgogICAgICAgIGNvbnRlbnRfbGVuZ3RoID0gc2VsZi5oZWFkZXJzLmdldCgiQ29udGVudC1MZW5ndGgiKQogICAgICAgIGRhdGEgPSBzZWxmLnJmaWxlLnJlYWQoaW50KGNvbnRlbnRfbGVuZ3RoKSkgaWYgY29udGVudF9sZW5ndGggZWxzZSBOb25lCiAgICAgICAgciA9IHJlcXVlc3QoCiAgICAgICAgICAgIHNlbGYuY29tbWFuZCwKICAgICAgICAgICAgZiJodHRwOi8ve1VQU1RSRUFNX0hPU1R9e3NlbGYucGF0aH0iLAogICAgICAgICAgICBoZWFkZXJzPXNlbGYuaGVhZGVycywKICAgICAgICAgICAgZGF0YT1kYXRhLAogICAgICAgICkKICAgICAgICBzZWxmLnNlbmRfcmVzcG9uc2Uoci5zdGF0dXNfY29kZSkKICAgICAgICBmb3IgaywgdiBpbiByLmhlYWRlcnMuaXRlbXMoKToKICAgICAgICAgICAgc2VsZi5zZW5kX2hlYWRlcihrLCB2KQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoci5jb250ZW50KQogICAgICAgIHNlbGYud2ZpbGUuY2xvc2UoKQoKCmNsYXNzIFRocmVhZGVkSFRUUFNlcnZlcihzb2NrZXRzZXJ2ZXIuVGhyZWFkaW5nTWl4SW4sIGh0dHAuc2VydmVyLkhUVFBTZXJ2ZXIpOgogICAgIiIiSGFuZGxlIHJlcXVlc3RzIGluIGEgc2VwYXJhdGUgdGhyZWFkLiIiIgoKCmh0dHBkID0gVGhyZWFkZWRIVFRQU2VydmVyKCgiMC4wLjAuMCIsIGludChzeXMuYXJndlsxXSkpLCBSZXF1ZXN0SGFuZGxlcikKaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > mocker_s3.py] Connecting to 172.16.5.2(172.16.5.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost run container_id:roottests3accessheaders-gw4-resolver-1 detach:True nothrow:False cmd: ['bash', '-c', 'python3 mocker_s3.py 8081 >/var/log/resolver/mocker_s3.log 2>/var/log/resolver/mocker_s3.err.log'] Command:[docker exec roottests3accessheaders-gw4-resolver-1 bash -c python3 mocker_s3.py 8081 >/var/log/resolver/mocker_s3.log 2>/var/log/resolver/mocker_s3.err.log] run container_id:roottests3accessheaders-gw4-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] Command:[docker exec roottests3accessheaders-gw4-resolver-1 curl -s http://localhost:8081/] Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestreplicatedzerocopyprojectionmutation-gw8-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.5.4, port:2181, use_ssl:False Connecting to 172.16.5.4(172.16.5.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Exitcode:7 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestreplicatedzerocopyprojectionmutation-gw8-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.5.3, port:2181, use_ssl:False Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') Trying to create Minio instance by command docker compose --project-name roottestreplicatedzerocopyprojectionmutation-gw8 --env-file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d Command:[docker compose --project-name roottestreplicatedzerocopyprojectionmutation-gw8 --env-file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d] Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=3828, time_out=30000, session_id=9, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=998, time_out=30000, session_id=7, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:1631 run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:772 Clickhouse process running. run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:772 Executing query select 20 on node Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 run container_id:roottests3accessheaders-gw4-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8081/'] Command:[docker exec roottests3accessheaders-gw4-resolver-1 curl -s http://localhost:8081/] Stderr:time="2025-04-02T03:59:03Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Volume "roottestreplicatedzerocopyprojectionmutation-gw8_data1-1" Creating Stderr: Volume "roottestreplicatedzerocopyprojectionmutation-gw8_data1-1" Created Stderr:time="2025-04-02T03:59:03Z" level=warning msg="Found orphan containers ([roottestreplicatedzerocopyprojectionmutation-gw8-zoo1-1 roottestreplicatedzerocopyprojectionmutation-gw8-zoo2-1 roottestreplicatedzerocopyprojectionmutation-gw8-zoo3-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up." Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy2-1 Creating Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy1-1 Creating Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy1-1 Created Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy2-1 Created Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-minio1-1 Creating Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 Creating Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 Created Executing query INSERT INTO test (key, value) VALUES (0, 'a'); SELECT * FROM test; on node Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-minio1-1 Created Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy2-1 Starting Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy1-1 Starting Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy2-1 Started Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy1-1 Started Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-minio1-1 Starting Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 Starting Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 Started Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-minio1-1 Started Stderr:time="2025-04-02T03:59:04Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:59:04Z" level=debug msg="otel error" error="" Trying to connect to Minio... get_instance_ip instance_name=minio1 http://localhost:None "GET /v1.46/containers/roottestreplicatedzerocopyprojectionmutation-gw8-minio1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=proxy1 http://localhost:None "GET /v1.46/containers/roottestreplicatedzerocopyprojectionmutation-gw8-proxy1-1/json HTTP/1.1" 200 None Starting new HTTP connection (1): 172.16.5.8:9001 Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (2): 172.16.5.8:9001 Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (3): 172.16.5.8:9001 Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (4): 172.16.5.8:9001 Can't connect to Minio: HTTPConnectionPool(host='172.16.5.8', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) Stdout:OK mocker_s3.py answered OK on attempt 2 Mock server mocker_s3.py started Executing query SET s3_truncate_on_insert=1; INSERT INTO FUNCTION s3('http://minio1:9001/root/test_static_override.csv', 'minio', 'minio123','CSV') SELECT number as a, toString(number) as b FROM numbers(3); on node1 Executing query DROP TABLE test; on node Executing query DROP TABLE IF EXISTS test_static_override; CREATE TABLE test_static_override (name String, value UInt32) ENGINE=S3('http://resolver:8081/root/test_static_override.csv', 'minio', 'minio123', 'CSV'); on node1 Executing query CREATE TABLE test (key UInt64, value String) Engine=EmbeddedRocksDB(0, '/var/lib/clickhouse/store/test_rocksdb_read_only_missing', 1) PRIMARY KEY(key); on node Executing query SYSTEM DROP QUERY CACHE on node1 run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrocksdbreadonly-gw3-node-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 772 ? 00:00:00 clickhouse run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrocksdbreadonly-gw3-node-1 bash -c pkill clickhouse] Executing query SELECT count(*) FROM test_static_override on node1 run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:772 run container_id:roottests3accessheaders-gw4-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/custom-auth-token: ValidToken1234/custom-auth-token: InvalidToken1234/g' /etc/clickhouse-server/config.d/s3_headers.xml"] Command:[docker exec roottests3accessheaders-gw4-node1-1 bash -c sed -i 's/custom-auth-token: ValidToken1234/custom-auth-token: InvalidToken1234/g' /etc/clickhouse-server/config.d/s3_headers.xml] Executing query SYSTEM RELOAD CONFIG on node1 Executing query SELECT count(*) FROM test_static_override on node1 Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 run container_id:roottests3accessheaders-gw4-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/custom-auth-token: InvalidToken1234/custom-auth-token: ValidToken1234/g' /etc/clickhouse-server/config.d/s3_headers.xml"] Command:[docker exec roottests3accessheaders-gw4-node1-1 bash -c sed -i 's/custom-auth-token: InvalidToken1234/custom-auth-token: ValidToken1234/g' /etc/clickhouse-server/config.d/s3_headers.xml] Executing query SYSTEM RELOAD CONFIG on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT count(*) FROM test_static_override on node1 http://localhost:None "GET /v1.46/exec/c5cf8ae238c3a8c70792b442d4d10aece49f23632167c1c825816c6755c92887/json HTTP/1.1" 200 584 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrefreshablemv-gw5-node1-1/exec HTTP/1.1" 201 74 [gw4] PASSED test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header] Executing query SET s3_truncate_on_insert=1; INSERT INTO FUNCTION s3('http://minio1:9001/root/test_access_header.csv', 'minio', 'minio123','CSV') SELECT number as a, toString(number) as b FROM numbers(3); on node1 test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header] http://localhost:None "POST /v1.46/exec/1f3b5214f823f0a0ea396a8bff8f3a98ae1434661385fa9f55a6441c68c142f3/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/1f3b5214f823f0a0ea396a8bff8f3a98ae1434661385fa9f55a6441c68c142f3/json HTTP/1.1" 200 586 Executing query DROP TABLE IF EXISTS test_access_header; CREATE TABLE test_access_header (name String, value UInt32) ENGINE=S3('http://resolver:8081/root/test_access_header.csv', 'CSV'); on node1 Executing query SYSTEM DROP QUERY CACHE on node1 Executing query SELECT count(*) FROM test_access_header on node1 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Starting new HTTP connection (5): 172.16.5.8:9001 http://172.16.5.8:9001 "GET / HTTP/1.1" 200 0 Connected to Minio. http://172.16.5.8:9001 "GET /root?location= HTTP/1.1" 404 0 http://172.16.5.8:9001 "PUT /root HTTP/1.1" 200 0 S3 bucket 'root' created http://172.16.5.8:9001 "GET /root2?location= HTTP/1.1" 404 0 http://172.16.5.8:9001 "PUT /root2 HTTP/1.1" 200 0 S3 bucket 'root2' created ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/.env --project-name roottestreplicatedzerocopyprojectionmutation-gw8 --file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/.env --project-name roottestreplicatedzerocopyprojectionmutation-gw8 --file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node2/docker-compose.yml up -d --no-recreate] run container_id:roottests3accessheaders-gw4-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/custom-auth-token: ValidToken1234/custom-auth-token: InvalidToken1234/g' /etc/clickhouse-server/config.d/s3_headers.xml"] Command:[docker exec roottests3accessheaders-gw4-node1-1 bash -c sed -i 's/custom-auth-token: ValidToken1234/custom-auth-token: InvalidToken1234/g' /etc/clickhouse-server/config.d/s3_headers.xml] Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 Executing query SELECT * FROM viewIfPermitted(SELECT x1 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x2 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x3 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x4 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x5 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x6 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x7 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x8 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x9 AS c FROM tbl ELSE null('c Int64')) UNION ALL SELECT * FROM viewIfPermitted(SELECT x10 AS c FROM tbl ELSE null('c Int64')) on instance Executing query SYSTEM RELOAD CONFIG on node1 Executing query SELECT count(*) FROM test_access_header on node1 Executing query DROP USER A, B, C on instance run container_id:roottests3accessheaders-gw4-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/custom-auth-token: InvalidToken1234/custom-auth-token: ValidToken1234/g' /etc/clickhouse-server/config.d/s3_headers.xml"] Command:[docker exec roottests3accessheaders-gw4-node1-1 bash -c sed -i 's/custom-auth-token: InvalidToken1234/custom-auth-token: ValidToken1234/g' /etc/clickhouse-server/config.d/s3_headers.xml] Executing query DROP ROLE R2, R1 on instance Executing query SYSTEM RELOAD CONFIG on node1 Executing query DROP TABLE tbl on instance run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query DROP USER IF EXISTS A, B on instance Stdout:772 [gw0] PASSED test_role/test.py::test_roles_cache Executing query SELECT count(*) FROM test_access_header on node1 Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo2-1 Running Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy2-1 Running Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo1-1 Running Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo3-1 Running Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy1-1 Running Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-minio1-1 Running Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 Creating Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 Running Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 Creating Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 Created Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 Created Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 Starting Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 Starting Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 Started Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestreplicatedzerocopyprojectionmutation-gw8-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestreplicatedzerocopyprojectionmutation-gw8-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.5.9... http://localhost:None "GET /v1.46/containers/roottestreplicatedzerocopyprojectionmutation-gw8-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/1c23dbfe852ae11476d672a19f03d10151ae20eff279e66750f78d7f0cb99117/json HTTP/1.1" 200 None Executing query DROP ROLE IF EXISTS R1, R2, R3, R4 on instance [gw4] PASSED test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header] test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header] Executing query SET s3_truncate_on_insert=1; INSERT INTO FUNCTION s3('http://minio1:9001/root/test_named_colections.csv', 'minio', 'minio123','CSV') SELECT number as a, toString(number) as b FROM numbers(3); on node1 Executing query CREATE USER A on instance test_role/test.py::test_set_role Executing query DROP TABLE IF EXISTS test_named_colections; CREATE TABLE test_named_colections (name String, value UInt32) ENGINE=S3(s3_mock, format='CSV'); on node1 http://localhost:None "GET /v1.46/containers/1c23dbfe852ae11476d672a19f03d10151ae20eff279e66750f78d7f0cb99117/json HTTP/1.1" 200 None Executing query CREATE ROLE R1, R2 on instance Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 Executing query SYSTEM DROP QUERY CACHE on node1 Executing query GRANT R1, R2 TO A on instance http://localhost:None "GET /v1.46/containers/1c23dbfe852ae11476d672a19f03d10151ae20eff279e66750f78d7f0cb99117/json HTTP/1.1" 200 None run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT count(*) FROM test_named_colections on node1 Executing query SHOW CURRENT ROLES on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%234&query=SHOW+CURRENT+ROLES HTTP/1.1" 200 None Executing query SET ROLE R1 on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%234&query=SET+ROLE+R1 HTTP/1.1" 200 None Executing query SHOW CURRENT ROLES on instance via HTTP interface Stdout:2457 Clickhouse process running. run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%234&query=SHOW+CURRENT+ROLES HTTP/1.1" 200 None Executing query SET ROLE R2 on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%234&query=SET+ROLE+R2 HTTP/1.1" 200 None Executing query SHOW CURRENT ROLES on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%234&query=SHOW+CURRENT+ROLES HTTP/1.1" 200 None Executing query SET ROLE NONE on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%234&query=SET+ROLE+NONE HTTP/1.1" 200 None Executing query SHOW CURRENT ROLES on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%234&query=SHOW+CURRENT+ROLES HTTP/1.1" 200 None Executing query SET ROLE DEFAULT on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 http://172.16.1.2:8123 "GET /?session_id=session+%234&query=SET+ROLE+DEFAULT HTTP/1.1" 200 None Executing query SHOW CURRENT ROLES on instance via HTTP interface Starting new HTTP connection (1): 172.16.1.2:8123 run container_id:roottests3accessheaders-gw4-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/custom-auth-token: ValidToken1234/custom-auth-token: InvalidToken1234/g' /etc/clickhouse-server/config.d/s3_headers.xml"] Command:[docker exec roottests3accessheaders-gw4-node1-1 bash -c sed -i 's/custom-auth-token: ValidToken1234/custom-auth-token: InvalidToken1234/g' /etc/clickhouse-server/config.d/s3_headers.xml] http://172.16.1.2:8123 "GET /?session_id=session+%234&query=SHOW+CURRENT+ROLES HTTP/1.1" 200 None [gw0] PASSED test_role/test.py::test_set_role Executing query DROP USER IF EXISTS A, B on instance http://localhost:None "GET /v1.46/containers/1c23dbfe852ae11476d672a19f03d10151ae20eff279e66750f78d7f0cb99117/json HTTP/1.1" 200 None Stdout:2457 Executing query select 20 on node1 Executing query SYSTEM RELOAD CONFIG on node1 Executing query DROP ROLE IF EXISTS R1, R2, R3, R4 on instance run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c ps -C clickhouse] Executing query SELECT count(*) FROM test_named_colections on node1 http://localhost:None "GET /v1.46/containers/1c23dbfe852ae11476d672a19f03d10151ae20eff279e66750f78d7f0cb99117/json HTTP/1.1" 200 None Command:[docker compose --env-file /ClickHouse/tests/integration/test_role/_instances-0-gw0/.env --project-name roottestrole-gw0 --file /ClickHouse/tests/integration/test_role/_instances-0-gw0/instance/docker-compose.yml stop --timeout 20] Stdout: PID TTY TIME CMD Stdout: 2457 ? 00:00:00 clickhouse run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c pkill clickhouse] run container_id:roottests3accessheaders-gw4-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "sed -i 's/custom-auth-token: InvalidToken1234/custom-auth-token: ValidToken1234/g' /etc/clickhouse-server/config.d/s3_headers.xml"] Command:[docker exec roottests3accessheaders-gw4-node1-1 bash -c sed -i 's/custom-auth-token: InvalidToken1234/custom-auth-token: ValidToken1234/g' /etc/clickhouse-server/config.d/s3_headers.xml] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/containers/1c23dbfe852ae11476d672a19f03d10151ae20eff279e66750f78d7f0cb99117/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 Executing query SYSTEM RELOAD CONFIG on node1 http://localhost:None "GET /v1.46/containers/roottestreplicatedzerocopyprojectionmutation-gw8-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestreplicatedzerocopyprojectionmutation-gw8-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.5.10... http://localhost:None "GET /v1.46/containers/roottestreplicatedzerocopyprojectionmutation-gw8-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4e5adad4862ad4e432164df64697940e2d2956a5ed10793bf9a90d40de3f8d48/json HTTP/1.1" 200 None ClickHouse node2 started Starting mock server broken_s3.py run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname broken_s3.py) && echo aW1wb3J0IGh0dHAuc2VydmVyCmltcG9ydCBsb2dnaW5nCmltcG9ydCByYW5kb20KaW1wb3J0IHNvY2tldAppbXBvcnQgc29ja2V0c2VydmVyCmltcG9ydCBzdHJpbmcKaW1wb3J0IHN0cnVjdAppbXBvcnQgc3lzCmltcG9ydCB0aHJlYWRpbmcKaW1wb3J0IHRpbWUKaW1wb3J0IHVybGxpYi5wYXJzZQoKSU5GX0NPVU5UID0gMTAwMDAwMDAwCgoKZGVmIF9hbmRfdGhlbih2YWx1ZSwgZnVuYyk6CiAgICBhc3NlcnQgY2FsbGFibGUoZnVuYykKICAgIHJldHVybiBOb25lIGlmIHZhbHVlIGlzIE5vbmUgZWxzZSBmdW5jKHZhbHVlKQoKCmNsYXNzIE1vY2tDb250cm9sOgogICAgZGVmIF9faW5pdF9fKHNlbGYsIGNsdXN0ZXIsIGNvbnRhaW5lciwgcG9ydCk6CiAgICAgICAgc2VsZi5fY2x1c3RlciA9IGNsdXN0ZXIKICAgICAgICBzZWxmLl9jb250YWluZXIgPSBjb250YWluZXIKICAgICAgICBzZWxmLl9wb3J0ID0gcG9ydAoKICAgIGRlZiByZXNldChzZWxmKToKICAgICAgICByZXNwb25zZSA9IHNlbGYuX2NsdXN0ZXIuZXhlY19pbl9jb250YWluZXIoCiAgICAgICAgICAgIHNlbGYuX2NsdXN0ZXIuZ2V0X2NvbnRhaW5lcl9pZChzZWxmLl9jb250YWluZXIpLAogICAgICAgICAgICBbCiAgICAgICAgICAgICAgICAiY3VybCIsCiAgICAgICAgICAgICAgICAiLXMiLAogICAgICAgICAgICAgICAgZiJodHRwOi8vbG9jYWxob3N0OntzZWxmLl9wb3J0fS9tb2NrX3NldHRpbmdzL3Jlc2V0IiwKICAgICAgICAgICAgXSwKICAgICAgICAgICAgbm90aHJvdz1UcnVlLAogICAgICAgICkKICAgICAgICBhc3NlcnQgcmVzcG9uc2UgPT0gIk9LIiwgcmVzcG9uc2UKCiAgICBkZWYgc2V0dXBfYWN0aW9uKHNlbGYsIHdoZW4sIGNvdW50PU5vbmUsIGFmdGVyPU5vbmUsIGFjdGlvbj1Ob25lLCBhY3Rpb25fYXJncz1Ob25lKToKICAgICAgICB1cmwgPSBmImh0dHA6Ly9sb2NhbGhvc3Q6e3NlbGYuX3BvcnR9L21vY2tfc2V0dGluZ3Mve3doZW59P25vdGhpbmc9MSIKCiAgICAgICAgaWYgY291bnQgaXMgbm90IE5vbmU6CiAgICAgICAgICAgIHVybCArPSBmIiZjb3VudD17Y291bnR9IgoKICAgICAgICBpZiBhZnRlciBpcyBub3QgTm9uZToKICAgICAgICAgICAgdXJsICs9IGYiJmFmdGVyPXthZnRlcn0iCgogICAgICAgIGlmIGFjdGlvbiBpcyBub3QgTm9uZToKICAgICAgICAgICAgdXJsICs9IGYiJmFjdGlvbj17YWN0aW9ufSIKCiAgICAgICAgaWYgYWN0aW9uX2FyZ3MgaXMgbm90IE5vbmU6CiAgICAgICAgICAgIGZvciB4IGluIGFjdGlvbl9hcmdzOgogICAgICAgICAgICAgICAgdXJsICs9IGYiJmFjdGlvbl9hcmdzPXt4fSIKCiAgICAgICAgcmVzcG9uc2UgPSBzZWxmLl9jbHVzdGVyLmV4ZWNfaW5fY29udGFpbmVyKAogICAgICAgICAgICBzZWxmLl9jbHVzdGVyLmdldF9jb250YWluZXJfaWQoc2VsZi5fY29udGFpbmVyKSwKICAgICAgICAgICAgWwogICAgICAgICAgICAgICAgImN1cmwiLAogICAgICAgICAgICAgICAgIi1zIiwKICAgICAgICAgICAgICAgIHVybCwKICAgICAgICAgICAgXSwKICAgICAgICAgICAgbm90aHJvdz1UcnVlLAogICAgICAgICkKICAgICAgICBhc3NlcnQgcmVzcG9uc2UgPT0gIk9LIiwgcmVzcG9uc2UKCiAgICBkZWYgc2V0dXBfYXRfb2JqZWN0X3VwbG9hZChzZWxmLCAqKmt3YXJncyk6CiAgICAgICAgc2VsZi5zZXR1cF9hY3Rpb24oImF0X29iamVjdF91cGxvYWQiLCAqKmt3YXJncykKCiAgICBkZWYgc2V0dXBfYXRfcGFydF91cGxvYWQoc2VsZiwgKiprd2FyZ3MpOgogICAgICAgIHNlbGYuc2V0dXBfYWN0aW9uKCJhdF9wYXJ0X3VwbG9hZCIsICoqa3dhcmdzKQoKICAgIGRlZiBzZXR1cF9hdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQoc2VsZiwgKiprd2FyZ3MpOgogICAgICAgIHNlbGYuc2V0dXBfYWN0aW9uKCJhdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQiLCAqKmt3YXJncykKCiAgICBkZWYgc2V0dXBfZmFrZV9wdXRzKHNlbGYsIHBhcnRfbGVuZ3RoKToKICAgICAgICByZXNwb25zZSA9IHNlbGYuX2NsdXN0ZXIuZXhlY19pbl9jb250YWluZXIoCiAgICAgICAgICAgIHNlbGYuX2NsdXN0ZXIuZ2V0X2NvbnRhaW5lcl9pZChzZWxmLl9jb250YWluZXIpLAogICAgICAgICAgICBbCiAgICAgICAgICAgICAgICAiY3VybCIsCiAgICAgICAgICAgICAgICAiLXMiLAogICAgICAgICAgICAgICAgZiJodHRwOi8vbG9jYWxob3N0OntzZWxmLl9wb3J0fS9tb2NrX3NldHRpbmdzL2Zha2VfcHV0cz93aGVuX2xlbmd0aF9iaWdnZXI9e3BhcnRfbGVuZ3RofSIsCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgIG5vdGhyb3c9VHJ1ZSwKICAgICAgICApCiAgICAgICAgYXNzZXJ0IHJlc3BvbnNlID09ICJPSyIsIHJlc3BvbnNlCgogICAgZGVmIHNldHVwX2Zha2VfbXVsdHBhcnR1cGxvYWRzKHNlbGYpOgogICAgICAgIHJlc3BvbnNlID0gc2VsZi5fY2x1c3Rlci5leGVjX2luX2NvbnRhaW5lcigKICAgICAgICAgICAgc2VsZi5fY2x1c3Rlci5nZXRfY29udGFpbmVyX2lkKHNlbGYuX2NvbnRhaW5lciksCiAgICAgICAgICAgIFsKICAgICAgICAgICAgICAgICJjdXJsIiwKICAgICAgICAgICAgICAgICItcyIsCiAgICAgICAgICAgICAgICBmImh0dHA6Ly9sb2NhbGhvc3Q6e3NlbGYuX3BvcnR9L21vY2tfc2V0dGluZ3Mvc2V0dXBfZmFrZV9tdWx0cGFydHVwbG9hZHM/IiwKICAgICAgICAgICAgXSwKICAgICAgICAgICAgbm90aHJvdz1UcnVlLAogICAgICAgICkKICAgICAgICBhc3NlcnQgcmVzcG9uc2UgPT0gIk9LIiwgcmVzcG9uc2UKCiAgICBkZWYgc2V0dXBfc2xvd19hbnN3ZXJzKAogICAgICAgIHNlbGYsIG1pbmltYWxfbGVuZ3RoPTAsIHRpbWVvdXQ9Tm9uZSwgcHJvYmFiaWxpdHk9Tm9uZSwgY291bnQ9Tm9uZQogICAgKToKICAgICAgICB1cmwgPSAoCiAgICAgICAgICAgIGYiaHR0cDovL2xvY2FsaG9zdDp7c2VsZi5fcG9ydH0vIgogICAgICAgICAgICBmIm1vY2tfc2V0dGluZ3Mvc2xvd19wdXQiCiAgICAgICAgICAgIGYiP21pbmltYWxfbGVuZ3RoPXttaW5pbWFsX2xlbmd0aH0iCiAgICAgICAgKQoKICAgICAgICBpZiB0aW1lb3V0IGlzIG5vdCBOb25lOgogICAgICAgICAgICB1cmwgKz0gZiImdGltZW91dD17dGltZW91dH0iCgogICAgICAgIGlmIHByb2JhYmlsaXR5IGlzIG5vdCBOb25lOgogICAgICAgICAgICB1cmwgKz0gZiImcHJvYmFiaWxpdHk9e3Byb2JhYmlsaXR5fSIKCiAgICAgICAgaWYgY291bnQgaXMgbm90IE5vbmU6CiAgICAgICAgICAgIHVybCArPSBmIiZjb3VudD17Y291bnR9IgoKICAgICAgICByZXNwb25zZSA9IHNlbGYuX2NsdXN0ZXIuZXhlY19pbl9jb250YWluZXIoCiAgICAgICAgICAgIHNlbGYuX2NsdXN0ZXIuZ2V0X2NvbnRhaW5lcl9pZChzZWxmLl9jb250YWluZXIpLAogICAgICAgICAgICBbImN1cmwiLCAiLXMiLCB1cmxdLAogICAgICAgICAgICBub3Rocm93PVRydWUsCiAgICAgICAgKQogICAgICAgIGFzc2VydCByZXNwb25zZSA9PSAiT0siLCByZXNwb25zZQoKCmNsYXNzIF9TZXJ2ZXJSdW50aW1lOgogICAgY2xhc3MgU2xvd1B1dDoKICAgICAgICBkZWYgX19pbml0X18oCiAgICAgICAgICAgIHNlbGYsCiAgICAgICAgICAgIGxvY2ssCiAgICAgICAgICAgIHByb2JhYmlsaXR5Xz1Ob25lLAogICAgICAgICAgICB0aW1lb3V0Xz1Ob25lLAogICAgICAgICAgICBtaW5pbWFsX2xlbmd0aF89Tm9uZSwKICAgICAgICAgICAgY291bnRfPU5vbmUsCiAgICAgICAgKToKICAgICAgICAgICAgc2VsZi5sb2NrID0gbG9jawogICAgICAgICAgICBzZWxmLnByb2JhYmlsaXR5ID0gcHJvYmFiaWxpdHlfIGlmIHByb2JhYmlsaXR5XyBpcyBub3QgTm9uZSBlbHNlIDEKICAgICAgICAgICAgc2VsZi50aW1lb3V0ID0gdGltZW91dF8gaWYgdGltZW91dF8gaXMgbm90IE5vbmUgZWxzZSAwLjEKICAgICAgICAgICAgc2VsZi5taW5pbWFsX2xlbmd0aCA9IG1pbmltYWxfbGVuZ3RoXyBpZiBtaW5pbWFsX2xlbmd0aF8gaXMgbm90IE5vbmUgZWxzZSAwCiAgICAgICAgICAgIHNlbGYuY291bnQgPSBjb3VudF8gaWYgY291bnRfIGlzIG5vdCBOb25lIGVsc2UgSU5GX0NPVU5UCgogICAgICAgIGRlZiBfX3N0cl9fKHNlbGYpOgogICAgICAgICAgICByZXR1cm4gKAogICAgICAgICAgICAgICAgZiJwcm9iYWJpbGl0eTp7c2VsZi5wcm9iYWJpbGl0eX0iCiAgICAgICAgICAgICAgICBmIiB0aW1lb3V0OntzZWxmLnRpbWVvdXR9IgogICAgICAgICAgICAgICAgZiIgbWluaW1hbF9sZW5ndGg6e3NlbGYubWluaW1hbF9sZW5ndGh9IgogICAgICAgICAgICAgICAgZiIgY291bnQ6e3NlbGYuY291bnR9IgogICAgICAgICAgICApCgogICAgICAgIGRlZiBnZXRfdGltZW91dChzZWxmLCBjb250ZW50X2xlbmd0aCk6CiAgICAgICAgICAgIHdpdGggc2VsZi5sb2NrOgogICAgICAgICAgICAgICAgaWYgY29udGVudF9sZW5ndGggPiBzZWxmLm1pbmltYWxfbGVuZ3RoOgogICAgICAgICAgICAgICAgICAgIGlmIHNlbGYuY291bnQgPiAwOgogICAgICAgICAgICAgICAgICAgICAgICBpZiAoCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBfcnVudGltZS5zbG93X3B1dC5wcm9iYWJpbGl0eSA9PSAxCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBvciByYW5kb20ucmFuZG9tKCkgPD0gX3J1bnRpbWUuc2xvd19wdXQucHJvYmFiaWxpdHkKICAgICAgICAgICAgICAgICAgICAgICAgKToKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHNlbGYuY291bnQgLT0gMQogICAgICAgICAgICAgICAgICAgICAgICAgICAgcmV0dXJuIF9ydW50aW1lLnNsb3dfcHV0LnRpbWVvdXQKICAgICAgICAgICAgcmV0dXJuIE5vbmUKCiAgICBjbGFzcyBFeHBlY3RlZDUwMEVycm9yQWN0aW9uOgogICAgICAgIGRlZiBpbmplY3RfZXJyb3Ioc2VsZiwgcmVxdWVzdF9oYW5kbGVyKToKICAgICAgICAgICAgZGF0YSA9ICgKICAgICAgICAgICAgICAgICc8P3htbCB2ZXJzaW9uPSIxLjAiIGVuY29kaW5nPSJVVEYtOCI/PicKICAgICAgICAgICAgICAgICI8RXJyb3I+IgogICAgICAgICAgICAgICAgIjxDb2RlPkV4cGVjdGVkRXJyb3I8L0NvZGU+IgogICAgICAgICAgICAgICAgIjxNZXNzYWdlPm1vY2sgczMgaW5qZWN0ZWQgdW5yZXRyeWFibGUgZXJyb3I8L01lc3NhZ2U+IgogICAgICAgICAgICAgICAgIjxSZXF1ZXN0SWQ+dHhmYmQ1NjZkMDMwNDI0NzQ4ODgxOTMtMDA2MDhkNzUzNzwvUmVxdWVzdElkPiIKICAgICAgICAgICAgICAgICI8L0Vycm9yPiIKICAgICAgICAgICAgKQogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIud3JpdGVfZXJyb3IoNTAwLCBkYXRhKQoKICAgIGNsYXNzIFNsb3dEb3duQWN0aW9uOgogICAgICAgIGRlZiBpbmplY3RfZXJyb3Ioc2VsZiwgcmVxdWVzdF9oYW5kbGVyKToKICAgICAgICAgICAgZGF0YSA9ICgKICAgICAgICAgICAgICAgICc8P3htbCB2ZXJzaW9uPSIxLjAiIGVuY29kaW5nPSJVVEYtOCI/PicKICAgICAgICAgICAgICAgICI8RXJyb3I+IgogICAgICAgICAgICAgICAgIjxDb2RlPlNsb3dEb3duPC9Db2RlPiIKICAgICAgICAgICAgICAgICI8TWVzc2FnZT5TbG93IERvd24uPC9NZXNzYWdlPiIKICAgICAgICAgICAgICAgICI8UmVxdWVzdElkPnR4ZmJkNTY2ZDAzMDQyNDc0ODg4MTkzLTAwNjA4ZDc1Mzc8L1JlcXVlc3RJZD4iCiAgICAgICAgICAgICAgICAiPC9FcnJvcj4iCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLndyaXRlX2Vycm9yKDQyOSwgZGF0YSkKCiAgICAjIG1ha2Ugc3VyZSB0aGF0IEFsaWJhYmEgZXJyb3JzIChRcHNMaW1pdEV4Y2VlZGVkLCBUb3RhbFFwc0xpbWl0RXhjZWVkZWRBY3Rpb24pIGFyZSByZXRyaWFibGUKICAgICMgd2UgcGF0Y2hlZCBjb250cmliL2F3cyB0byBhY2hpdmUgaXQ6IGh0dHBzOi8vZ2l0aHViLmNvbS9DbGlja0hvdXNlL2F3cy1zZGstY3BwL3B1bGwvMjIgaHR0cHM6Ly9naXRodWIuY29tL0NsaWNrSG91c2UvYXdzLXNkay1jcHAvcHVsbC8yMwogICAgIyBodHRwczovL3d3dy5hbGliYWJhY2xvdWQuY29tL2hlbHAvZW4vb3NzL3N1cHBvcnQvaHR0cC1zdGF0dXMtY29kZS01MDMKICAgIGNsYXNzIFFwc0xpbWl0RXhjZWVkZWRBY3Rpb246CiAgICAgICAgZGVmIGluamVjdF9lcnJvcihzZWxmLCByZXF1ZXN0X2hhbmRsZXIpOgogICAgICAgICAgICBkYXRhID0gKAogICAgICAgICAgICAgICAgJzw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+JwogICAgICAgICAgICAgICAgIjxFcnJvcj4iCiAgICAgICAgICAgICAgICAiPENvZGU+UXBzTGltaXRFeGNlZWRlZDwvQ29kZT4iCiAgICAgICAgICAgICAgICAiPE1lc3NhZ2U+UGxlYXNlIHJlZHVjZSB5b3VyIHJlcXVlc3QgcmF0ZS48L01lc3NhZ2U+IgogICAgICAgICAgICAgICAgIjxSZXF1ZXN0SWQ+dHhmYmQ1NjZkMDMwNDI0NzQ4ODgxOTMtMDA2MDhkNzUzNzwvUmVxdWVzdElkPiIKICAgICAgICAgICAgICAgICI8L0Vycm9yPiIKICAgICAgICAgICAgKQogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIud3JpdGVfZXJyb3IoNDI5LCBkYXRhKQoKICAgIGNsYXNzIFRvdGFsUXBzTGltaXRFeGNlZWRlZEFjdGlvbjoKICAgICAgICBkZWYgaW5qZWN0X2Vycm9yKHNlbGYsIHJlcXVlc3RfaGFuZGxlcik6CiAgICAgICAgICAgIGRhdGEgPSAoCiAgICAgICAgICAgICAgICAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4nCiAgICAgICAgICAgICAgICAiPEVycm9yPiIKICAgICAgICAgICAgICAgICI8Q29kZT5Ub3RhbFFwc0xpbWl0RXhjZWVkZWQ8L0NvZGU+IgogICAgICAgICAgICAgICAgIjxNZXNzYWdlPlBsZWFzZSByZWR1Y2UgeW91ciByZXF1ZXN0IHJhdGUuPC9NZXNzYWdlPiIKICAgICAgICAgICAgICAgICI8UmVxdWVzdElkPnR4ZmJkNTY2ZDAzMDQyNDc0ODg4MTkzLTAwNjA4ZDc1Mzc8L1JlcXVlc3RJZD4iCiAgICAgICAgICAgICAgICAiPC9FcnJvcj4iCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLndyaXRlX2Vycm9yKDQyOSwgZGF0YSkKCiAgICBjbGFzcyBSZWRpcmVjdEFjdGlvbjoKICAgICAgICBkZWYgX19pbml0X18oc2VsZiwgaG9zdD0ibG9jYWxob3N0IiwgcG9ydD0xKToKICAgICAgICAgICAgc2VsZi5kc3RfaG9zdCA9IF9hbmRfdGhlbihob3N0LCBzdHIpCiAgICAgICAgICAgIHNlbGYuZHN0X3BvcnQgPSBfYW5kX3RoZW4ocG9ydCwgaW50KQoKICAgICAgICBkZWYgaW5qZWN0X2Vycm9yKHNlbGYsIHJlcXVlc3RfaGFuZGxlcik6CiAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5yZWRpcmVjdChob3N0PXNlbGYuZHN0X2hvc3QsIHBvcnQ9c2VsZi5kc3RfcG9ydCkKCiAgICBjbGFzcyBDb25uZWN0aW9uUmVzZXRCeVBlZXJBY3Rpb246CiAgICAgICAgZGVmIF9faW5pdF9fKHNlbGYsIHdpdGhfcGFydGlhbF9kYXRhPU5vbmUpOgogICAgICAgICAgICBzZWxmLnBhcnRpYWxfZGF0YSA9ICIiCiAgICAgICAgICAgIGlmIHdpdGhfcGFydGlhbF9kYXRhIGlzIG5vdCBOb25lIGFuZCB3aXRoX3BhcnRpYWxfZGF0YSA9PSAiMSI6CiAgICAgICAgICAgICAgICBzZWxmLnBhcnRpYWxfZGF0YSA9ICgKICAgICAgICAgICAgICAgICAgICAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz5cbicKICAgICAgICAgICAgICAgICAgICAiPEluaXRpYXRlTXVsdGlwYXJ0VXBsb2FkUmVzdWx0PlxuIgogICAgICAgICAgICAgICAgKQoKICAgICAgICBkZWYgaW5qZWN0X2Vycm9yKHNlbGYsIHJlcXVlc3RfaGFuZGxlcik6CiAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5yZWFkX2FsbF9pbnB1dCgpCgogICAgICAgICAgICBpZiBzZWxmLnBhcnRpYWxfZGF0YToKICAgICAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5zZW5kX3Jlc3BvbnNlKDIwMCkKICAgICAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQveG1sIikKICAgICAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCAxMDAwMCkKICAgICAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5lbmRfaGVhZGVycygpCiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIud2ZpbGUud3JpdGUoYnl0ZXMoc2VsZi5wYXJ0aWFsX2RhdGEsICJVVEYtOCIpKQoKICAgICAgICAgICAgdGltZS5zbGVlcCgxKQogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuY29ubmVjdGlvbi5zZXRzb2Nrb3B0KAogICAgICAgICAgICAgICAgc29ja2V0LlNPTF9TT0NLRVQsIHNvY2tldC5TT19MSU5HRVIsIHN0cnVjdC5wYWNrKCJpaSIsIDEsIDApCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLmNvbm5lY3Rpb24uY2xvc2UoKQoKICAgIGNsYXNzIEJyb2tlblBpcGVBY3Rpb246CiAgICAgICAgZGVmIGluamVjdF9lcnJvcihzZWxmLCByZXF1ZXN0X2hhbmRsZXIpOgogICAgICAgICAgICAjIHBhcnRpYWwgcmVhZAogICAgICAgICAgICBzZWxmLnJmaWxlLnJlYWQoNTApCgogICAgICAgICAgICB0aW1lLnNsZWVwKDEpCiAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5jb25uZWN0aW9uLnNldHNvY2tvcHQoCiAgICAgICAgICAgICAgICBzb2NrZXQuU09MX1NPQ0tFVCwgc29ja2V0LlNPX0xJTkdFUiwgc3RydWN0LnBhY2soImlpIiwgMSwgMCkKICAgICAgICAgICAgKQogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuY29ubmVjdGlvbi5jbG9zZSgpCgogICAgY2xhc3MgQ29ubmVjdGlvblJlZnVzZWRBY3Rpb24oUmVkaXJlY3RBY3Rpb24pOgogICAgICAgIHBhc3MKCiAgICBjbGFzcyBDb3VudEFmdGVyOgogICAgICAgIGRlZiBfX2luaXRfXygKICAgICAgICAgICAgc2VsZiwgbG9jaywgY291bnRfPU5vbmUsIGFmdGVyXz1Ob25lLCBhY3Rpb25fPU5vbmUsIGFjdGlvbl9hcmdzXz1bXQogICAgICAgICk6CiAgICAgICAgICAgIHNlbGYubG9jayA9IGxvY2sKCiAgICAgICAgICAgIHNlbGYuY291bnQgPSBjb3VudF8gaWYgY291bnRfIGlzIG5vdCBOb25lIGVsc2UgSU5GX0NPVU5UCiAgICAgICAgICAgIHNlbGYuYWZ0ZXIgPSBhZnRlcl8gaWYgYWZ0ZXJfIGlzIG5vdCBOb25lIGVsc2UgMAogICAgICAgICAgICBzZWxmLmFjdGlvbiA9IGFjdGlvbl8KICAgICAgICAgICAgc2VsZi5hY3Rpb25fYXJncyA9IGFjdGlvbl9hcmdzXwoKICAgICAgICAgICAgaWYgc2VsZi5hY3Rpb24gPT0gImNvbm5lY3Rpb25fcmVmdXNlZCI6CiAgICAgICAgICAgICAgICBzZWxmLmVycm9yX2hhbmRsZXIgPSBfU2VydmVyUnVudGltZS5Db25uZWN0aW9uUmVmdXNlZEFjdGlvbigpCiAgICAgICAgICAgIGVsaWYgc2VsZi5hY3Rpb24gPT0gImNvbm5lY3Rpb25fcmVzZXRfYnlfcGVlciI6CiAgICAgICAgICAgICAgICBzZWxmLmVycm9yX2hhbmRsZXIgPSBfU2VydmVyUnVudGltZS5Db25uZWN0aW9uUmVzZXRCeVBlZXJBY3Rpb24oCiAgICAgICAgICAgICAgICAgICAgKnNlbGYuYWN0aW9uX2FyZ3MKICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgZWxpZiBzZWxmLmFjdGlvbiA9PSAiYnJva2VuX3BpcGUiOgogICAgICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyID0gX1NlcnZlclJ1bnRpbWUuQnJva2VuUGlwZUFjdGlvbigpCiAgICAgICAgICAgIGVsaWYgc2VsZi5hY3Rpb24gPT0gInJlZGlyZWN0X3RvIjoKICAgICAgICAgICAgICAgIHNlbGYuZXJyb3JfaGFuZGxlciA9IF9TZXJ2ZXJSdW50aW1lLlJlZGlyZWN0QWN0aW9uKCpzZWxmLmFjdGlvbl9hcmdzKQogICAgICAgICAgICBlbGlmIHNlbGYuYWN0aW9uID09ICJzbG93X2Rvd24iOgogICAgICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyID0gX1NlcnZlclJ1bnRpbWUuU2xvd0Rvd25BY3Rpb24oKnNlbGYuYWN0aW9uX2FyZ3MpCiAgICAgICAgICAgIGVsaWYgc2VsZi5hY3Rpb24gPT0gInFwc19saW1pdF9leGNlZWRlZCI6CiAgICAgICAgICAgICAgICBzZWxmLmVycm9yX2hhbmRsZXIgPSBfU2VydmVyUnVudGltZS5RcHNMaW1pdEV4Y2VlZGVkQWN0aW9uKAogICAgICAgICAgICAgICAgICAgICpzZWxmLmFjdGlvbl9hcmdzCiAgICAgICAgICAgICAgICApCiAgICAgICAgICAgIGVsaWYgc2VsZi5hY3Rpb24gPT0gInRvdGFsX3Fwc19saW1pdF9leGNlZWRlZCI6CiAgICAgICAgICAgICAgICBzZWxmLmVycm9yX2hhbmRsZXIgPSBfU2VydmVyUnVudGltZS5Ub3RhbFFwc0xpbWl0RXhjZWVkZWRBY3Rpb24oCiAgICAgICAgICAgICAgICAgICAgKnNlbGYuYWN0aW9uX2FyZ3MKICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgZWxzZToKICAgICAgICAgICAgICAgIHNlbGYuZXJyb3JfaGFuZGxlciA9IF9TZXJ2ZXJSdW50aW1lLkV4cGVjdGVkNTAwRXJyb3JBY3Rpb24oKQoKICAgICAgICBAc3RhdGljbWV0aG9kCiAgICAgICAgZGVmIGZyb21fY2dpX3BhcmFtcyhsb2NrLCBwYXJhbXMpOgogICAgICAgICAgICByZXR1cm4gX1NlcnZlclJ1bnRpbWUuQ291bnRBZnRlcigKICAgICAgICAgICAgICAgIGxvY2s9bG9jaywKICAgICAgICAgICAgICAgIGNvdW50Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgiY291bnQiLCBbTm9uZV0pWzBdLCBpbnQpLAogICAgICAgICAgICAgICAgYWZ0ZXJfPV9hbmRfdGhlbihwYXJhbXMuZ2V0KCJhZnRlciIsIFtOb25lXSlbMF0sIGludCksCiAgICAgICAgICAgICAgICBhY3Rpb25fPXBhcmFtcy5nZXQoImFjdGlvbiIsIFtOb25lXSlbMF0sCiAgICAgICAgICAgICAgICBhY3Rpb25fYXJnc189cGFyYW1zLmdldCgiYWN0aW9uX2FyZ3MiLCBbXSksCiAgICAgICAgICAgICkKCiAgICAgICAgZGVmIF9fc3RyX18oc2VsZik6CiAgICAgICAgICAgIHJldHVybiBmImNvdW50OntzZWxmLmNvdW50fSBhZnRlcjp7c2VsZi5hZnRlcn0gYWN0aW9uOntzZWxmLmFjdGlvbn0gYWN0aW9uX2FyZ3M6e3NlbGYuYWN0aW9uX2FyZ3N9IgoKICAgICAgICBkZWYgaGFzX2VmZmVjdChzZWxmKToKICAgICAgICAgICAgd2l0aCBzZWxmLmxvY2s6CiAgICAgICAgICAgICAgICBpZiBzZWxmLmFmdGVyOgogICAgICAgICAgICAgICAgICAgIHNlbGYuYWZ0ZXIgLT0gMQogICAgICAgICAgICAgICAgaWYgc2VsZi5hZnRlciA9PSAwOgogICAgICAgICAgICAgICAgICAgIGlmIHNlbGYuY291bnQ6CiAgICAgICAgICAgICAgICAgICAgICAgIHNlbGYuY291bnQgLT0gMQogICAgICAgICAgICAgICAgICAgICAgICByZXR1cm4gVHJ1ZQogICAgICAgICAgICAgICAgcmV0dXJuIEZhbHNlCgogICAgICAgIGRlZiBpbmplY3RfZXJyb3Ioc2VsZiwgcmVxdWVzdF9oYW5kbGVyKToKICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyLmluamVjdF9lcnJvcihyZXF1ZXN0X2hhbmRsZXIpCgogICAgZGVmIF9faW5pdF9fKHNlbGYpOgogICAgICAgIHNlbGYubG9jayA9IHRocmVhZGluZy5Mb2NrKCkKICAgICAgICBzZWxmLmF0X3BhcnRfdXBsb2FkID0gTm9uZQogICAgICAgIHNlbGYuYXRfb2JqZWN0X3VwbG9hZCA9IE5vbmUKICAgICAgICBzZWxmLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IE5vbmUKICAgICAgICBzZWxmLmZha2VfdXBsb2FkcyA9IGRpY3QoKQogICAgICAgIHNlbGYuc2xvd19wdXQgPSBOb25lCiAgICAgICAgc2VsZi5mYWtlX211bHRpcGFydF91cGxvYWQgPSBOb25lCiAgICAgICAgc2VsZi5hdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQgPSBOb25lCgogICAgZGVmIHJlZ2lzdGVyX2Zha2VfdXBsb2FkKHNlbGYsIHVwbG9hZF9pZCwga2V5KToKICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgc2VsZi5mYWtlX3VwbG9hZHNbdXBsb2FkX2lkXSA9IGtleQoKICAgIGRlZiBpc19mYWtlX3VwbG9hZChzZWxmLCB1cGxvYWRfaWQsIGtleSk6CiAgICAgICAgd2l0aCBzZWxmLmxvY2s6CiAgICAgICAgICAgIGlmIHVwbG9hZF9pZCBpbiBzZWxmLmZha2VfdXBsb2FkczoKICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLmZha2VfdXBsb2Fkc1t1cGxvYWRfaWRdID09IGtleQogICAgICAgIHJldHVybiBGYWxzZQoKICAgIGRlZiByZXNldChzZWxmKToKICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgc2VsZi5hdF9wYXJ0X3VwbG9hZCA9IE5vbmUKICAgICAgICAgICAgc2VsZi5hdF9vYmplY3RfdXBsb2FkID0gTm9uZQogICAgICAgICAgICBzZWxmLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IE5vbmUKICAgICAgICAgICAgc2VsZi5mYWtlX3VwbG9hZHMgPSBkaWN0KCkKICAgICAgICAgICAgc2VsZi5zbG93X3B1dCA9IE5vbmUKICAgICAgICAgICAgc2VsZi5mYWtlX211bHRpcGFydF91cGxvYWQgPSBOb25lCiAgICAgICAgICAgIHNlbGYuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkID0gTm9uZQoKCl9ydW50aW1lID0gX1NlcnZlclJ1bnRpbWUoKQoKCmRlZiBnZXRfcmFuZG9tX3N0cmluZyhsZW5ndGgpOgogICAgIyBjaG9vc2UgZnJvbSBhbGwgbG93ZXJjYXNlIGxldHRlcgogICAgbGV0dGVycyA9IHN0cmluZy5hc2NpaV9sb3dlcmNhc2UKICAgIHJlc3VsdF9zdHIgPSAiIi5qb2luKHJhbmRvbS5jaG9pY2UobGV0dGVycykgZm9yIGkgaW4gcmFuZ2UobGVuZ3RoKSkKICAgIHJldHVybiByZXN1bHRfc3RyCgoKY2xhc3MgUmVxdWVzdEhhbmRsZXIoaHR0cC5zZXJ2ZXIuQmFzZUhUVFBSZXF1ZXN0SGFuZGxlcik6CiAgICBkZWYgX29rKHNlbGYpOgogICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQvcGxhaW4iKQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYiJPSyIpCgogICAgZGVmIF9waW5nKHNlbGYpOgogICAgICAgIHNlbGYuX29rKCkKCiAgICBkZWYgcmVhZF9hbGxfaW5wdXQoc2VsZik6CiAgICAgICAgY29udGVudF9sZW5ndGggPSBpbnQoc2VsZi5oZWFkZXJzLmdldCgiQ29udGVudC1MZW5ndGgiLCAwKSkKICAgICAgICB0b19yZWFkID0gY29udGVudF9sZW5ndGgKICAgICAgICB3aGlsZSB0b19yZWFkID4gMDoKICAgICAgICAgICAgIyByZWFkIGNvbnRlbnQgaW4gb3JkZXIgdG8gYXZvaWQgZXJyb3Igb24gY2xpZW50CiAgICAgICAgICAgICMgUG9jbzo6RXhjZXB0aW9uLiBDb2RlOiAxMDAwLCBlLmNvZGUoKSA9IDMyLCBJL08gZXJyb3I6IEJyb2tlbiBwaXBlCiAgICAgICAgICAgICMgZG8gaXQgcGllY2UgYnkgcGllY2UgaW4gb3JkZXIgdG8gYXZvaWQgYmlnIGFsbG9jYXRpb24KICAgICAgICAgICAgc2l6ZSA9IG1pbih0b19yZWFkLCAxMDI0KQogICAgICAgICAgICBzdHIoc2VsZi5yZmlsZS5yZWFkKHNpemUpKQogICAgICAgICAgICB0b19yZWFkIC09IHNpemUKCiAgICBkZWYgcmVkaXJlY3Qoc2VsZiwgaG9zdD1Ob25lLCBwb3J0PU5vbmUpOgogICAgICAgIGlmIGhvc3QgaXMgTm9uZSBhbmQgcG9ydCBpcyBOb25lOgogICAgICAgICAgICBob3N0ID0gc2VsZi5zZXJ2ZXIudXBzdHJlYW1faG9zdAogICAgICAgICAgICBwb3J0ID0gc2VsZi5zZXJ2ZXIudXBzdHJlYW1fcG9ydAoKICAgICAgICBzZWxmLnJlYWRfYWxsX2lucHV0KCkKCiAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDMwNykKICAgICAgICB1cmwgPSBmImh0dHA6Ly97aG9zdH06e3BvcnR9e3NlbGYucGF0aH0iCiAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgicmVkaXJlY3QgdG8gJXMiLCB1cmwpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiTG9jYXRpb24iLCB1cmwpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCiAgICAgICAgc2VsZi53ZmlsZS53cml0ZShiIlJlZGlyZWN0ZWQiKQoKICAgIGRlZiB3cml0ZV9lcnJvcihzZWxmLCBodHRwX2NvZGUsIGRhdGEsIGNvbnRlbnRfbGVuZ3RoPU5vbmUpOgogICAgICAgIGlmIGNvbnRlbnRfbGVuZ3RoIGlzIE5vbmU6CiAgICAgICAgICAgIGNvbnRlbnRfbGVuZ3RoID0gbGVuKGRhdGEpCiAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgid3JpdGVfZXJyb3IgJXMiLCBkYXRhKQogICAgICAgIHNlbGYucmVhZF9hbGxfaW5wdXQoKQogICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZShodHRwX2NvZGUpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQveG1sIikKICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIHN0cihjb250ZW50X2xlbmd0aCkpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCiAgICAgICAgaWYgZGF0YToKICAgICAgICAgICAgc2VsZi53ZmlsZS53cml0ZShieXRlcyhkYXRhLCAiVVRGLTgiKSkKCiAgICBkZWYgX2Zha2VfcHV0X29rKHNlbGYpOgogICAgICAgIHNlbGYubG9nX21lc3NhZ2UoImZha2UgcHV0IikKCiAgICAgICAgc2VsZi5yZWFkX2FsbF9pbnB1dCgpCgogICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQveG1sIikKICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJFVGFnIiwgImI1NDM1N2ZhZjA2MzJjY2U0NmU5NDJmYTY4MzU2YjM4IikKICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIDApCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgZGVmIF9mYWtlX3VwbG9hZHMoc2VsZiwgcGF0aCwgdXBsb2FkX2lkKToKICAgICAgICBzZWxmLnJlYWRfYWxsX2lucHV0KCkKCiAgICAgICAgcGFydHMgPSBbeCBmb3IgeCBpbiBwYXRoLnNwbGl0KCIvIikgaWYgeF0KICAgICAgICBidWNrZXQgPSBwYXJ0c1swXQogICAgICAgIGtleSA9ICIvIi5qb2luKHBhcnRzWzE6XSkKICAgICAgICBkYXRhID0gKAogICAgICAgICAgICAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz5cbicKICAgICAgICAgICAgIjxJbml0aWF0ZU11bHRpcGFydFVwbG9hZFJlc3VsdD5cbiIKICAgICAgICAgICAgZiI8QnVja2V0PntidWNrZXR9PC9CdWNrZXQ+IgogICAgICAgICAgICBmIjxLZXk+e2tleX08L0tleT4iCiAgICAgICAgICAgIGYiPFVwbG9hZElkPnt1cGxvYWRfaWR9PC9VcGxvYWRJZD4iCiAgICAgICAgICAgICI8L0luaXRpYXRlTXVsdGlwYXJ0VXBsb2FkUmVzdWx0PiIKICAgICAgICApCgogICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQveG1sIikKICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIGxlbihkYXRhKSkKICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKCiAgICAgICAgc2VsZi53ZmlsZS53cml0ZShieXRlcyhkYXRhLCAiVVRGLTgiKSkKCiAgICBkZWYgX2Zha2VfcG9zdF9vayhzZWxmLCBwYXRoKToKICAgICAgICBzZWxmLnJlYWRfYWxsX2lucHV0KCkKCiAgICAgICAgcGFydHMgPSBbeCBmb3IgeCBpbiBwYXRoLnNwbGl0KCIvIikgaWYgeF0KICAgICAgICBidWNrZXQgPSBwYXJ0c1swXQogICAgICAgIGtleSA9ICIvIi5qb2luKHBhcnRzWzE6XSkKICAgICAgICBsb2NhdGlvbiA9ICJodHRwOi8vRXhhbXBsZS1CdWNrZXQuczMuUmVnaW9uLmFtYXpvbmF3cy5jb20vIiArIHBhdGgKICAgICAgICBkYXRhID0gKAogICAgICAgICAgICAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz5cbicKICAgICAgICAgICAgIjxDb21wbGV0ZU11bHRpcGFydFVwbG9hZFJlc3VsdD5cbiIKICAgICAgICAgICAgZiI8TG9jYXRpb24+e2xvY2F0aW9ufTwvTG9jYXRpb24+XG4iCiAgICAgICAgICAgIGYiPEJ1Y2tldD57YnVja2V0fTwvQnVja2V0PlxuIgogICAgICAgICAgICBmIjxLZXk+e2tleX08L0tleT5cbiIKICAgICAgICAgICAgZic8RVRhZz4iMzg1OGY2MjIzMGFjM2M5MTVmMzAwYzY2NDMxMmMxMWYtOSI8L0VUYWc+XG4nCiAgICAgICAgICAgIGYiPC9Db21wbGV0ZU11bHRpcGFydFVwbG9hZFJlc3VsdD5cbiIKICAgICAgICApCgogICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQveG1sIikKICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIGxlbihkYXRhKSkKICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKCiAgICAgICAgc2VsZi53ZmlsZS53cml0ZShieXRlcyhkYXRhLCAiVVRGLTgiKSkKCiAgICBkZWYgX21vY2tfc2V0dGluZ3Moc2VsZik6CiAgICAgICAgcGFydHMgPSB1cmxsaWIucGFyc2UudXJsc3BsaXQoc2VsZi5wYXRoKQogICAgICAgIHBhdGggPSBbeCBmb3IgeCBpbiBwYXJ0cy5wYXRoLnNwbGl0KCIvIikgaWYgeF0KICAgICAgICBhc3NlcnQgcGF0aFswXSA9PSAibW9ja19zZXR0aW5ncyIsIHBhdGgKICAgICAgICBpZiBsZW4ocGF0aCkgPCAyOgogICAgICAgICAgICByZXR1cm4gc2VsZi53cml0ZV9lcnJvcig0MDAsICJfbW9ja19zZXR0aW5nczogd3JvbmcgY29tbWFuZCIpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gImF0X3BhcnRfdXBsb2FkIjoKICAgICAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICAgICAgX3J1bnRpbWUuYXRfcGFydF91cGxvYWQgPSBfU2VydmVyUnVudGltZS5Db3VudEFmdGVyLmZyb21fY2dpX3BhcmFtcygKICAgICAgICAgICAgICAgIF9ydW50aW1lLmxvY2ssIHBhcmFtcwogICAgICAgICAgICApCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoInNldCBhdF9wYXJ0X3VwbG9hZCAlcyIsIF9ydW50aW1lLmF0X3BhcnRfdXBsb2FkKQogICAgICAgICAgICByZXR1cm4gc2VsZi5fb2soKQoKICAgICAgICBpZiBwYXRoWzFdID09ICJhdF9vYmplY3RfdXBsb2FkIjoKICAgICAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICAgICAgX3J1bnRpbWUuYXRfb2JqZWN0X3VwbG9hZCA9IF9TZXJ2ZXJSdW50aW1lLkNvdW50QWZ0ZXIuZnJvbV9jZ2lfcGFyYW1zKAogICAgICAgICAgICAgICAgX3J1bnRpbWUubG9jaywgcGFyYW1zCiAgICAgICAgICAgICkKICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2V0IGF0X29iamVjdF91cGxvYWQgJXMiLCBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkKQogICAgICAgICAgICByZXR1cm4gc2VsZi5fb2soKQoKICAgICAgICBpZiBwYXRoWzFdID09ICJmYWtlX3B1dHMiOgogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPUZhbHNlKQogICAgICAgICAgICBfcnVudGltZS5mYWtlX3B1dF93aGVuX2xlbmd0aF9iaWdnZXIgPSBpbnQoCiAgICAgICAgICAgICAgICBwYXJhbXMuZ2V0KCJ3aGVuX2xlbmd0aF9iaWdnZXIiLCBbMTAyNCAqIDEwMjRdKVswXQogICAgICAgICAgICApCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoInNldCBmYWtlX3B1dHMgJXMiLCBfcnVudGltZS5mYWtlX3B1dF93aGVuX2xlbmd0aF9iaWdnZXIpCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gInNsb3dfcHV0IjoKICAgICAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICAgICAgX3J1bnRpbWUuc2xvd19wdXQgPSBfU2VydmVyUnVudGltZS5TbG93UHV0KAogICAgICAgICAgICAgICAgbG9jaz1fcnVudGltZS5sb2NrLAogICAgICAgICAgICAgICAgbWluaW1hbF9sZW5ndGhfPV9hbmRfdGhlbihwYXJhbXMuZ2V0KCJtaW5pbWFsX2xlbmd0aCIsIFtOb25lXSlbMF0sIGludCksCiAgICAgICAgICAgICAgICBwcm9iYWJpbGl0eV89X2FuZF90aGVuKHBhcmFtcy5nZXQoInByb2JhYmlsaXR5IiwgW05vbmVdKVswXSwgZmxvYXQpLAogICAgICAgICAgICAgICAgdGltZW91dF89X2FuZF90aGVuKHBhcmFtcy5nZXQoInRpbWVvdXQiLCBbTm9uZV0pWzBdLCBmbG9hdCksCiAgICAgICAgICAgICAgICBjb3VudF89X2FuZF90aGVuKHBhcmFtcy5nZXQoImNvdW50IiwgW05vbmVdKVswXSwgaW50KSwKICAgICAgICAgICAgKQogICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKCJzZXQgc2xvdyBwdXQgJXMiLCBfcnVudGltZS5zbG93X3B1dCkKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAic2V0dXBfZmFrZV9tdWx0cGFydHVwbG9hZHMiOgogICAgICAgICAgICBfcnVudGltZS5mYWtlX211bHRpcGFydF91cGxvYWQgPSBUcnVlCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoInNldCBzZXR1cF9mYWtlX211bHRwYXJ0dXBsb2FkcyIpCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gImF0X2NyZWF0ZV9tdWx0aV9wYXJ0X3VwbG9hZCI6CiAgICAgICAgICAgIHBhcmFtcyA9IHVybGxpYi5wYXJzZS5wYXJzZV9xcyhwYXJ0cy5xdWVyeSwga2VlcF9ibGFua192YWx1ZXM9RmFsc2UpCiAgICAgICAgICAgIF9ydW50aW1lLmF0X2NyZWF0ZV9tdWx0aV9wYXJ0X3VwbG9hZCA9ICgKICAgICAgICAgICAgICAgIF9TZXJ2ZXJSdW50aW1lLkNvdW50QWZ0ZXIuZnJvbV9jZ2lfcGFyYW1zKF9ydW50aW1lLmxvY2ssIHBhcmFtcykKICAgICAgICAgICAgKQogICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKAogICAgICAgICAgICAgICAgInNldCBhdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQgJXMiLAogICAgICAgICAgICAgICAgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkLAogICAgICAgICAgICApCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gInJlc2V0IjoKICAgICAgICAgICAgX3J1bnRpbWUucmVzZXQoKQogICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKCJyZXNldCIpCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIHJldHVybiBzZWxmLndyaXRlX2Vycm9yKDQwMCwgIl9tb2NrX3NldHRpbmdzOiB3cm9uZyBjb21tYW5kIikKCiAgICBkZWYgZG9fR0VUKHNlbGYpOgogICAgICAgIGlmIHNlbGYucGF0aCA9PSAiLyI6CiAgICAgICAgICAgIHJldHVybiBzZWxmLl9waW5nKCkKCiAgICAgICAgaWYgc2VsZi5wYXRoLnN0YXJ0c3dpdGgoIi9tb2NrX3NldHRpbmdzIik6CiAgICAgICAgICAgIHJldHVybiBzZWxmLl9tb2NrX3NldHRpbmdzKCkKCiAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgiZ2V0IHJlZGlyZWN0IikKICAgICAgICByZXR1cm4gc2VsZi5yZWRpcmVjdCgpCgogICAgZGVmIGRvX1BVVChzZWxmKToKICAgICAgICBjb250ZW50X2xlbmd0aCA9IGludChzZWxmLmhlYWRlcnMuZ2V0KCJDb250ZW50LUxlbmd0aCIsIDApKQoKICAgICAgICBpZiBfcnVudGltZS5zbG93X3B1dCBpcyBub3QgTm9uZToKICAgICAgICAgICAgdGltZW91dCA9IF9ydW50aW1lLnNsb3dfcHV0LmdldF90aW1lb3V0KGNvbnRlbnRfbGVuZ3RoKQogICAgICAgICAgICBpZiB0aW1lb3V0IGlzIG5vdCBOb25lOgogICAgICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2xvdyBwdXQgJXMiLCB0aW1lb3V0KQogICAgICAgICAgICAgICAgdGltZS5zbGVlcCh0aW1lb3V0KQoKICAgICAgICBwYXJ0cyA9IHVybGxpYi5wYXJzZS51cmxzcGxpdChzZWxmLnBhdGgpCiAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICB1cGxvYWRfaWQgPSBwYXJhbXMuZ2V0KCJ1cGxvYWRJZCIsIFtOb25lXSlbMF0KCiAgICAgICAgaWYgdXBsb2FkX2lkIGlzIG5vdCBOb25lOgogICAgICAgICAgICBpZiBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCBpcyBub3QgTm9uZToKICAgICAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICAgICAgICAgInB1dCBhdF9wYXJ0X3VwbG9hZCAlcywgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCwKICAgICAgICAgICAgICAgICAgICB1cGxvYWRfaWQsCiAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICApCgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfcGFydF91cGxvYWQuaGFzX2VmZmVjdCgpOgogICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZC5pbmplY3RfZXJyb3Ioc2VsZikKICAgICAgICAgICAgaWYgX3J1bnRpbWUuZmFrZV9tdWx0aXBhcnRfdXBsb2FkOgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuaXNfZmFrZV91cGxvYWQodXBsb2FkX2lkLCBwYXJ0cy5wYXRoKToKICAgICAgICAgICAgICAgICAgICByZXR1cm4gc2VsZi5fZmFrZV9wdXRfb2soKQogICAgICAgIGVsc2U6CiAgICAgICAgICAgIGlmIF9ydW50aW1lLmF0X29iamVjdF91cGxvYWQgaXMgbm90IE5vbmU6CiAgICAgICAgICAgICAgICBpZiBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkLmhhc19lZmZlY3QoKToKICAgICAgICAgICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKAogICAgICAgICAgICAgICAgICAgICAgICAicHV0IGVycm9yX2F0X29iamVjdF91cGxvYWQgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuYXRfb2JqZWN0X3VwbG9hZCwKICAgICAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkLmluamVjdF9lcnJvcihzZWxmKQogICAgICAgICAgICBpZiBfcnVudGltZS5mYWtlX3B1dF93aGVuX2xlbmd0aF9iaWdnZXIgaXMgbm90IE5vbmU6CiAgICAgICAgICAgICAgICBpZiBjb250ZW50X2xlbmd0aCA+IF9ydW50aW1lLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlcjoKICAgICAgICAgICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKAogICAgICAgICAgICAgICAgICAgICAgICAicHV0IGZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciAlcywgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuZmFrZV9wdXRfd2hlbl9sZW5ndGhfYmlnZ2VyLAogICAgICAgICAgICAgICAgICAgICAgICBjb250ZW50X2xlbmd0aCwKICAgICAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLl9mYWtlX3B1dF9vaygpCgogICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICJwdXQgcmVkaXJlY3QgJXMiLAogICAgICAgICAgICBwYXJ0cywKICAgICAgICApCiAgICAgICAgcmV0dXJuIHNlbGYucmVkaXJlY3QoKQoKICAgIGRlZiBkb19QT1NUKHNlbGYpOgogICAgICAgIHBhcnRzID0gdXJsbGliLnBhcnNlLnVybHNwbGl0KHNlbGYucGF0aCkKICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPVRydWUpCiAgICAgICAgdXBsb2FkcyA9IHBhcmFtcy5nZXQoInVwbG9hZHMiLCBbTm9uZV0pWzBdCiAgICAgICAgaWYgdXBsb2FkcyBpcyBub3QgTm9uZToKICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkIGlzIG5vdCBOb25lOgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkLmhhc19lZmZlY3QoKToKICAgICAgICAgICAgICAgICAgICByZXR1cm4gX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkLmluamVjdF9lcnJvcihzZWxmKQoKICAgICAgICAgICAgaWYgX3J1bnRpbWUuZmFrZV9tdWx0aXBhcnRfdXBsb2FkOgogICAgICAgICAgICAgICAgdXBsb2FkX2lkID0gZ2V0X3JhbmRvbV9zdHJpbmcoNSkKICAgICAgICAgICAgICAgIF9ydW50aW1lLnJlZ2lzdGVyX2Zha2VfdXBsb2FkKHVwbG9hZF9pZCwgcGFydHMucGF0aCkKICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLl9mYWtlX3VwbG9hZHMocGFydHMucGF0aCwgdXBsb2FkX2lkKQoKICAgICAgICB1cGxvYWRfaWQgPSBwYXJhbXMuZ2V0KCJ1cGxvYWRJZCIsIFtOb25lXSlbMF0KICAgICAgICBpZiBfcnVudGltZS5pc19mYWtlX3VwbG9hZCh1cGxvYWRfaWQsIHBhcnRzLnBhdGgpOgogICAgICAgICAgICByZXR1cm4gc2VsZi5fZmFrZV9wb3N0X29rKHBhcnRzLnBhdGgpCgogICAgICAgIHJldHVybiBzZWxmLnJlZGlyZWN0KCkKCiAgICBkZWYgZG9fSEVBRChzZWxmKToKICAgICAgICBzZWxmLnJlZGlyZWN0KCkKCiAgICBkZWYgZG9fREVMRVRFKHNlbGYpOgogICAgICAgIHNlbGYucmVkaXJlY3QoKQoKCmNsYXNzIF9UaHJlYWRlZEhUVFBTZXJ2ZXIoc29ja2V0c2VydmVyLlRocmVhZGluZ01peEluLCBodHRwLnNlcnZlci5IVFRQU2VydmVyKToKICAgICIiIkhhbmRsZSByZXF1ZXN0cyBpbiBhIHNlcGFyYXRlIHRocmVhZC4iIiIKCiAgICBkZWYgc2V0X3Vwc3RyZWFtKHNlbGYsIHVwc3RyZWFtX2hvc3QsIHVwc3RyZWFtX3BvcnQpOgogICAgICAgIHNlbGYudXBzdHJlYW1faG9zdCA9IHVwc3RyZWFtX2hvc3QKICAgICAgICBzZWxmLnVwc3RyZWFtX3BvcnQgPSB1cHN0cmVhbV9wb3J0CgoKaWYgX19uYW1lX18gPT0gIl9fbWFpbl9fIjoKICAgIGh0dHBkID0gX1RocmVhZGVkSFRUUFNlcnZlcigoIjAuMC4wLjAiLCBpbnQoc3lzLmFyZ3ZbMV0pKSwgUmVxdWVzdEhhbmRsZXIpCiAgICBpZiBsZW4oc3lzLmFyZ3YpID09IDQ6CiAgICAgICAgaHR0cGQuc2V0X3Vwc3RyZWFtKHN5cy5hcmd2WzJdLCBzeXMuYXJndlszXSkKICAgIGVsc2U6CiAgICAgICAgaHR0cGQuc2V0X3Vwc3RyZWFtKCJtaW5pbzEiLCA5MDAxKQogICAgaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > broken_s3.py'] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 bash -c mkdir -p $(dirname broken_s3.py) && echo aW1wb3J0IGh0dHAuc2VydmVyCmltcG9ydCBsb2dnaW5nCmltcG9ydCByYW5kb20KaW1wb3J0IHNvY2tldAppbXBvcnQgc29ja2V0c2VydmVyCmltcG9ydCBzdHJpbmcKaW1wb3J0IHN0cnVjdAppbXBvcnQgc3lzCmltcG9ydCB0aHJlYWRpbmcKaW1wb3J0IHRpbWUKaW1wb3J0IHVybGxpYi5wYXJzZQoKSU5GX0NPVU5UID0gMTAwMDAwMDAwCgoKZGVmIF9hbmRfdGhlbih2YWx1ZSwgZnVuYyk6CiAgICBhc3NlcnQgY2FsbGFibGUoZnVuYykKICAgIHJldHVybiBOb25lIGlmIHZhbHVlIGlzIE5vbmUgZWxzZSBmdW5jKHZhbHVlKQoKCmNsYXNzIE1vY2tDb250cm9sOgogICAgZGVmIF9faW5pdF9fKHNlbGYsIGNsdXN0ZXIsIGNvbnRhaW5lciwgcG9ydCk6CiAgICAgICAgc2VsZi5fY2x1c3RlciA9IGNsdXN0ZXIKICAgICAgICBzZWxmLl9jb250YWluZXIgPSBjb250YWluZXIKICAgICAgICBzZWxmLl9wb3J0ID0gcG9ydAoKICAgIGRlZiByZXNldChzZWxmKToKICAgICAgICByZXNwb25zZSA9IHNlbGYuX2NsdXN0ZXIuZXhlY19pbl9jb250YWluZXIoCiAgICAgICAgICAgIHNlbGYuX2NsdXN0ZXIuZ2V0X2NvbnRhaW5lcl9pZChzZWxmLl9jb250YWluZXIpLAogICAgICAgICAgICBbCiAgICAgICAgICAgICAgICAiY3VybCIsCiAgICAgICAgICAgICAgICAiLXMiLAogICAgICAgICAgICAgICAgZiJodHRwOi8vbG9jYWxob3N0OntzZWxmLl9wb3J0fS9tb2NrX3NldHRpbmdzL3Jlc2V0IiwKICAgICAgICAgICAgXSwKICAgICAgICAgICAgbm90aHJvdz1UcnVlLAogICAgICAgICkKICAgICAgICBhc3NlcnQgcmVzcG9uc2UgPT0gIk9LIiwgcmVzcG9uc2UKCiAgICBkZWYgc2V0dXBfYWN0aW9uKHNlbGYsIHdoZW4sIGNvdW50PU5vbmUsIGFmdGVyPU5vbmUsIGFjdGlvbj1Ob25lLCBhY3Rpb25fYXJncz1Ob25lKToKICAgICAgICB1cmwgPSBmImh0dHA6Ly9sb2NhbGhvc3Q6e3NlbGYuX3BvcnR9L21vY2tfc2V0dGluZ3Mve3doZW59P25vdGhpbmc9MSIKCiAgICAgICAgaWYgY291bnQgaXMgbm90IE5vbmU6CiAgICAgICAgICAgIHVybCArPSBmIiZjb3VudD17Y291bnR9IgoKICAgICAgICBpZiBhZnRlciBpcyBub3QgTm9uZToKICAgICAgICAgICAgdXJsICs9IGYiJmFmdGVyPXthZnRlcn0iCgogICAgICAgIGlmIGFjdGlvbiBpcyBub3QgTm9uZToKICAgICAgICAgICAgdXJsICs9IGYiJmFjdGlvbj17YWN0aW9ufSIKCiAgICAgICAgaWYgYWN0aW9uX2FyZ3MgaXMgbm90IE5vbmU6CiAgICAgICAgICAgIGZvciB4IGluIGFjdGlvbl9hcmdzOgogICAgICAgICAgICAgICAgdXJsICs9IGYiJmFjdGlvbl9hcmdzPXt4fSIKCiAgICAgICAgcmVzcG9uc2UgPSBzZWxmLl9jbHVzdGVyLmV4ZWNfaW5fY29udGFpbmVyKAogICAgICAgICAgICBzZWxmLl9jbHVzdGVyLmdldF9jb250YWluZXJfaWQoc2VsZi5fY29udGFpbmVyKSwKICAgICAgICAgICAgWwogICAgICAgICAgICAgICAgImN1cmwiLAogICAgICAgICAgICAgICAgIi1zIiwKICAgICAgICAgICAgICAgIHVybCwKICAgICAgICAgICAgXSwKICAgICAgICAgICAgbm90aHJvdz1UcnVlLAogICAgICAgICkKICAgICAgICBhc3NlcnQgcmVzcG9uc2UgPT0gIk9LIiwgcmVzcG9uc2UKCiAgICBkZWYgc2V0dXBfYXRfb2JqZWN0X3VwbG9hZChzZWxmLCAqKmt3YXJncyk6CiAgICAgICAgc2VsZi5zZXR1cF9hY3Rpb24oImF0X29iamVjdF91cGxvYWQiLCAqKmt3YXJncykKCiAgICBkZWYgc2V0dXBfYXRfcGFydF91cGxvYWQoc2VsZiwgKiprd2FyZ3MpOgogICAgICAgIHNlbGYuc2V0dXBfYWN0aW9uKCJhdF9wYXJ0X3VwbG9hZCIsICoqa3dhcmdzKQoKICAgIGRlZiBzZXR1cF9hdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQoc2VsZiwgKiprd2FyZ3MpOgogICAgICAgIHNlbGYuc2V0dXBfYWN0aW9uKCJhdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQiLCAqKmt3YXJncykKCiAgICBkZWYgc2V0dXBfZmFrZV9wdXRzKHNlbGYsIHBhcnRfbGVuZ3RoKToKICAgICAgICByZXNwb25zZSA9IHNlbGYuX2NsdXN0ZXIuZXhlY19pbl9jb250YWluZXIoCiAgICAgICAgICAgIHNlbGYuX2NsdXN0ZXIuZ2V0X2NvbnRhaW5lcl9pZChzZWxmLl9jb250YWluZXIpLAogICAgICAgICAgICBbCiAgICAgICAgICAgICAgICAiY3VybCIsCiAgICAgICAgICAgICAgICAiLXMiLAogICAgICAgICAgICAgICAgZiJodHRwOi8vbG9jYWxob3N0OntzZWxmLl9wb3J0fS9tb2NrX3NldHRpbmdzL2Zha2VfcHV0cz93aGVuX2xlbmd0aF9iaWdnZXI9e3BhcnRfbGVuZ3RofSIsCiAgICAgICAgICAgIF0sCiAgICAgICAgICAgIG5vdGhyb3c9VHJ1ZSwKICAgICAgICApCiAgICAgICAgYXNzZXJ0IHJlc3BvbnNlID09ICJPSyIsIHJlc3BvbnNlCgogICAgZGVmIHNldHVwX2Zha2VfbXVsdHBhcnR1cGxvYWRzKHNlbGYpOgogICAgICAgIHJlc3BvbnNlID0gc2VsZi5fY2x1c3Rlci5leGVjX2luX2NvbnRhaW5lcigKICAgICAgICAgICAgc2VsZi5fY2x1c3Rlci5nZXRfY29udGFpbmVyX2lkKHNlbGYuX2NvbnRhaW5lciksCiAgICAgICAgICAgIFsKICAgICAgICAgICAgICAgICJjdXJsIiwKICAgICAgICAgICAgICAgICItcyIsCiAgICAgICAgICAgICAgICBmImh0dHA6Ly9sb2NhbGhvc3Q6e3NlbGYuX3BvcnR9L21vY2tfc2V0dGluZ3Mvc2V0dXBfZmFrZV9tdWx0cGFydHVwbG9hZHM/IiwKICAgICAgICAgICAgXSwKICAgICAgICAgICAgbm90aHJvdz1UcnVlLAogICAgICAgICkKICAgICAgICBhc3NlcnQgcmVzcG9uc2UgPT0gIk9LIiwgcmVzcG9uc2UKCiAgICBkZWYgc2V0dXBfc2xvd19hbnN3ZXJzKAogICAgICAgIHNlbGYsIG1pbmltYWxfbGVuZ3RoPTAsIHRpbWVvdXQ9Tm9uZSwgcHJvYmFiaWxpdHk9Tm9uZSwgY291bnQ9Tm9uZQogICAgKToKICAgICAgICB1cmwgPSAoCiAgICAgICAgICAgIGYiaHR0cDovL2xvY2FsaG9zdDp7c2VsZi5fcG9ydH0vIgogICAgICAgICAgICBmIm1vY2tfc2V0dGluZ3Mvc2xvd19wdXQiCiAgICAgICAgICAgIGYiP21pbmltYWxfbGVuZ3RoPXttaW5pbWFsX2xlbmd0aH0iCiAgICAgICAgKQoKICAgICAgICBpZiB0aW1lb3V0IGlzIG5vdCBOb25lOgogICAgICAgICAgICB1cmwgKz0gZiImdGltZW91dD17dGltZW91dH0iCgogICAgICAgIGlmIHByb2JhYmlsaXR5IGlzIG5vdCBOb25lOgogICAgICAgICAgICB1cmwgKz0gZiImcHJvYmFiaWxpdHk9e3Byb2JhYmlsaXR5fSIKCiAgICAgICAgaWYgY291bnQgaXMgbm90IE5vbmU6CiAgICAgICAgICAgIHVybCArPSBmIiZjb3VudD17Y291bnR9IgoKICAgICAgICByZXNwb25zZSA9IHNlbGYuX2NsdXN0ZXIuZXhlY19pbl9jb250YWluZXIoCiAgICAgICAgICAgIHNlbGYuX2NsdXN0ZXIuZ2V0X2NvbnRhaW5lcl9pZChzZWxmLl9jb250YWluZXIpLAogICAgICAgICAgICBbImN1cmwiLCAiLXMiLCB1cmxdLAogICAgICAgICAgICBub3Rocm93PVRydWUsCiAgICAgICAgKQogICAgICAgIGFzc2VydCByZXNwb25zZSA9PSAiT0siLCByZXNwb25zZQoKCmNsYXNzIF9TZXJ2ZXJSdW50aW1lOgogICAgY2xhc3MgU2xvd1B1dDoKICAgICAgICBkZWYgX19pbml0X18oCiAgICAgICAgICAgIHNlbGYsCiAgICAgICAgICAgIGxvY2ssCiAgICAgICAgICAgIHByb2JhYmlsaXR5Xz1Ob25lLAogICAgICAgICAgICB0aW1lb3V0Xz1Ob25lLAogICAgICAgICAgICBtaW5pbWFsX2xlbmd0aF89Tm9uZSwKICAgICAgICAgICAgY291bnRfPU5vbmUsCiAgICAgICAgKToKICAgICAgICAgICAgc2VsZi5sb2NrID0gbG9jawogICAgICAgICAgICBzZWxmLnByb2JhYmlsaXR5ID0gcHJvYmFiaWxpdHlfIGlmIHByb2JhYmlsaXR5XyBpcyBub3QgTm9uZSBlbHNlIDEKICAgICAgICAgICAgc2VsZi50aW1lb3V0ID0gdGltZW91dF8gaWYgdGltZW91dF8gaXMgbm90IE5vbmUgZWxzZSAwLjEKICAgICAgICAgICAgc2VsZi5taW5pbWFsX2xlbmd0aCA9IG1pbmltYWxfbGVuZ3RoXyBpZiBtaW5pbWFsX2xlbmd0aF8gaXMgbm90IE5vbmUgZWxzZSAwCiAgICAgICAgICAgIHNlbGYuY291bnQgPSBjb3VudF8gaWYgY291bnRfIGlzIG5vdCBOb25lIGVsc2UgSU5GX0NPVU5UCgogICAgICAgIGRlZiBfX3N0cl9fKHNlbGYpOgogICAgICAgICAgICByZXR1cm4gKAogICAgICAgICAgICAgICAgZiJwcm9iYWJpbGl0eTp7c2VsZi5wcm9iYWJpbGl0eX0iCiAgICAgICAgICAgICAgICBmIiB0aW1lb3V0OntzZWxmLnRpbWVvdXR9IgogICAgICAgICAgICAgICAgZiIgbWluaW1hbF9sZW5ndGg6e3NlbGYubWluaW1hbF9sZW5ndGh9IgogICAgICAgICAgICAgICAgZiIgY291bnQ6e3NlbGYuY291bnR9IgogICAgICAgICAgICApCgogICAgICAgIGRlZiBnZXRfdGltZW91dChzZWxmLCBjb250ZW50X2xlbmd0aCk6CiAgICAgICAgICAgIHdpdGggc2VsZi5sb2NrOgogICAgICAgICAgICAgICAgaWYgY29udGVudF9sZW5ndGggPiBzZWxmLm1pbmltYWxfbGVuZ3RoOgogICAgICAgICAgICAgICAgICAgIGlmIHNlbGYuY291bnQgPiAwOgogICAgICAgICAgICAgICAgICAgICAgICBpZiAoCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBfcnVudGltZS5zbG93X3B1dC5wcm9iYWJpbGl0eSA9PSAxCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBvciByYW5kb20ucmFuZG9tKCkgPD0gX3J1bnRpbWUuc2xvd19wdXQucHJvYmFiaWxpdHkKICAgICAgICAgICAgICAgICAgICAgICAgKToKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHNlbGYuY291bnQgLT0gMQogICAgICAgICAgICAgICAgICAgICAgICAgICAgcmV0dXJuIF9ydW50aW1lLnNsb3dfcHV0LnRpbWVvdXQKICAgICAgICAgICAgcmV0dXJuIE5vbmUKCiAgICBjbGFzcyBFeHBlY3RlZDUwMEVycm9yQWN0aW9uOgogICAgICAgIGRlZiBpbmplY3RfZXJyb3Ioc2VsZiwgcmVxdWVzdF9oYW5kbGVyKToKICAgICAgICAgICAgZGF0YSA9ICgKICAgICAgICAgICAgICAgICc8P3htbCB2ZXJzaW9uPSIxLjAiIGVuY29kaW5nPSJVVEYtOCI/PicKICAgICAgICAgICAgICAgICI8RXJyb3I+IgogICAgICAgICAgICAgICAgIjxDb2RlPkV4cGVjdGVkRXJyb3I8L0NvZGU+IgogICAgICAgICAgICAgICAgIjxNZXNzYWdlPm1vY2sgczMgaW5qZWN0ZWQgdW5yZXRyeWFibGUgZXJyb3I8L01lc3NhZ2U+IgogICAgICAgICAgICAgICAgIjxSZXF1ZXN0SWQ+dHhmYmQ1NjZkMDMwNDI0NzQ4ODgxOTMtMDA2MDhkNzUzNzwvUmVxdWVzdElkPiIKICAgICAgICAgICAgICAgICI8L0Vycm9yPiIKICAgICAgICAgICAgKQogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIud3JpdGVfZXJyb3IoNTAwLCBkYXRhKQoKICAgIGNsYXNzIFNsb3dEb3duQWN0aW9uOgogICAgICAgIGRlZiBpbmplY3RfZXJyb3Ioc2VsZiwgcmVxdWVzdF9oYW5kbGVyKToKICAgICAgICAgICAgZGF0YSA9ICgKICAgICAgICAgICAgICAgICc8P3htbCB2ZXJzaW9uPSIxLjAiIGVuY29kaW5nPSJVVEYtOCI/PicKICAgICAgICAgICAgICAgICI8RXJyb3I+IgogICAgICAgICAgICAgICAgIjxDb2RlPlNsb3dEb3duPC9Db2RlPiIKICAgICAgICAgICAgICAgICI8TWVzc2FnZT5TbG93IERvd24uPC9NZXNzYWdlPiIKICAgICAgICAgICAgICAgICI8UmVxdWVzdElkPnR4ZmJkNTY2ZDAzMDQyNDc0ODg4MTkzLTAwNjA4ZDc1Mzc8L1JlcXVlc3RJZD4iCiAgICAgICAgICAgICAgICAiPC9FcnJvcj4iCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLndyaXRlX2Vycm9yKDQyOSwgZGF0YSkKCiAgICAjIG1ha2Ugc3VyZSB0aGF0IEFsaWJhYmEgZXJyb3JzIChRcHNMaW1pdEV4Y2VlZGVkLCBUb3RhbFFwc0xpbWl0RXhjZWVkZWRBY3Rpb24pIGFyZSByZXRyaWFibGUKICAgICMgd2UgcGF0Y2hlZCBjb250cmliL2F3cyB0byBhY2hpdmUgaXQ6IGh0dHBzOi8vZ2l0aHViLmNvbS9DbGlja0hvdXNlL2F3cy1zZGstY3BwL3B1bGwvMjIgaHR0cHM6Ly9naXRodWIuY29tL0NsaWNrSG91c2UvYXdzLXNkay1jcHAvcHVsbC8yMwogICAgIyBodHRwczovL3d3dy5hbGliYWJhY2xvdWQuY29tL2hlbHAvZW4vb3NzL3N1cHBvcnQvaHR0cC1zdGF0dXMtY29kZS01MDMKICAgIGNsYXNzIFFwc0xpbWl0RXhjZWVkZWRBY3Rpb246CiAgICAgICAgZGVmIGluamVjdF9lcnJvcihzZWxmLCByZXF1ZXN0X2hhbmRsZXIpOgogICAgICAgICAgICBkYXRhID0gKAogICAgICAgICAgICAgICAgJzw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+JwogICAgICAgICAgICAgICAgIjxFcnJvcj4iCiAgICAgICAgICAgICAgICAiPENvZGU+UXBzTGltaXRFeGNlZWRlZDwvQ29kZT4iCiAgICAgICAgICAgICAgICAiPE1lc3NhZ2U+UGxlYXNlIHJlZHVjZSB5b3VyIHJlcXVlc3QgcmF0ZS48L01lc3NhZ2U+IgogICAgICAgICAgICAgICAgIjxSZXF1ZXN0SWQ+dHhmYmQ1NjZkMDMwNDI0NzQ4ODgxOTMtMDA2MDhkNzUzNzwvUmVxdWVzdElkPiIKICAgICAgICAgICAgICAgICI8L0Vycm9yPiIKICAgICAgICAgICAgKQogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIud3JpdGVfZXJyb3IoNDI5LCBkYXRhKQoKICAgIGNsYXNzIFRvdGFsUXBzTGltaXRFeGNlZWRlZEFjdGlvbjoKICAgICAgICBkZWYgaW5qZWN0X2Vycm9yKHNlbGYsIHJlcXVlc3RfaGFuZGxlcik6CiAgICAgICAgICAgIGRhdGEgPSAoCiAgICAgICAgICAgICAgICAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4nCiAgICAgICAgICAgICAgICAiPEVycm9yPiIKICAgICAgICAgICAgICAgICI8Q29kZT5Ub3RhbFFwc0xpbWl0RXhjZWVkZWQ8L0NvZGU+IgogICAgICAgICAgICAgICAgIjxNZXNzYWdlPlBsZWFzZSByZWR1Y2UgeW91ciByZXF1ZXN0IHJhdGUuPC9NZXNzYWdlPiIKICAgICAgICAgICAgICAgICI8UmVxdWVzdElkPnR4ZmJkNTY2ZDAzMDQyNDc0ODg4MTkzLTAwNjA4ZDc1Mzc8L1JlcXVlc3RJZD4iCiAgICAgICAgICAgICAgICAiPC9FcnJvcj4iCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLndyaXRlX2Vycm9yKDQyOSwgZGF0YSkKCiAgICBjbGFzcyBSZWRpcmVjdEFjdGlvbjoKICAgICAgICBkZWYgX19pbml0X18oc2VsZiwgaG9zdD0ibG9jYWxob3N0IiwgcG9ydD0xKToKICAgICAgICAgICAgc2VsZi5kc3RfaG9zdCA9IF9hbmRfdGhlbihob3N0LCBzdHIpCiAgICAgICAgICAgIHNlbGYuZHN0X3BvcnQgPSBfYW5kX3RoZW4ocG9ydCwgaW50KQoKICAgICAgICBkZWYgaW5qZWN0X2Vycm9yKHNlbGYsIHJlcXVlc3RfaGFuZGxlcik6CiAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5yZWRpcmVjdChob3N0PXNlbGYuZHN0X2hvc3QsIHBvcnQ9c2VsZi5kc3RfcG9ydCkKCiAgICBjbGFzcyBDb25uZWN0aW9uUmVzZXRCeVBlZXJBY3Rpb246CiAgICAgICAgZGVmIF9faW5pdF9fKHNlbGYsIHdpdGhfcGFydGlhbF9kYXRhPU5vbmUpOgogICAgICAgICAgICBzZWxmLnBhcnRpYWxfZGF0YSA9ICIiCiAgICAgICAgICAgIGlmIHdpdGhfcGFydGlhbF9kYXRhIGlzIG5vdCBOb25lIGFuZCB3aXRoX3BhcnRpYWxfZGF0YSA9PSAiMSI6CiAgICAgICAgICAgICAgICBzZWxmLnBhcnRpYWxfZGF0YSA9ICgKICAgICAgICAgICAgICAgICAgICAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz5cbicKICAgICAgICAgICAgICAgICAgICAiPEluaXRpYXRlTXVsdGlwYXJ0VXBsb2FkUmVzdWx0PlxuIgogICAgICAgICAgICAgICAgKQoKICAgICAgICBkZWYgaW5qZWN0X2Vycm9yKHNlbGYsIHJlcXVlc3RfaGFuZGxlcik6CiAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5yZWFkX2FsbF9pbnB1dCgpCgogICAgICAgICAgICBpZiBzZWxmLnBhcnRpYWxfZGF0YToKICAgICAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5zZW5kX3Jlc3BvbnNlKDIwMCkKICAgICAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQveG1sIikKICAgICAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5zZW5kX2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCAxMDAwMCkKICAgICAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5lbmRfaGVhZGVycygpCiAgICAgICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIud2ZpbGUud3JpdGUoYnl0ZXMoc2VsZi5wYXJ0aWFsX2RhdGEsICJVVEYtOCIpKQoKICAgICAgICAgICAgdGltZS5zbGVlcCgxKQogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuY29ubmVjdGlvbi5zZXRzb2Nrb3B0KAogICAgICAgICAgICAgICAgc29ja2V0LlNPTF9TT0NLRVQsIHNvY2tldC5TT19MSU5HRVIsIHN0cnVjdC5wYWNrKCJpaSIsIDEsIDApCiAgICAgICAgICAgICkKICAgICAgICAgICAgcmVxdWVzdF9oYW5kbGVyLmNvbm5lY3Rpb24uY2xvc2UoKQoKICAgIGNsYXNzIEJyb2tlblBpcGVBY3Rpb246CiAgICAgICAgZGVmIGluamVjdF9lcnJvcihzZWxmLCByZXF1ZXN0X2hhbmRsZXIpOgogICAgICAgICAgICAjIHBhcnRpYWwgcmVhZAogICAgICAgICAgICBzZWxmLnJmaWxlLnJlYWQoNTApCgogICAgICAgICAgICB0aW1lLnNsZWVwKDEpCiAgICAgICAgICAgIHJlcXVlc3RfaGFuZGxlci5jb25uZWN0aW9uLnNldHNvY2tvcHQoCiAgICAgICAgICAgICAgICBzb2NrZXQuU09MX1NPQ0tFVCwgc29ja2V0LlNPX0xJTkdFUiwgc3RydWN0LnBhY2soImlpIiwgMSwgMCkKICAgICAgICAgICAgKQogICAgICAgICAgICByZXF1ZXN0X2hhbmRsZXIuY29ubmVjdGlvbi5jbG9zZSgpCgogICAgY2xhc3MgQ29ubmVjdGlvblJlZnVzZWRBY3Rpb24oUmVkaXJlY3RBY3Rpb24pOgogICAgICAgIHBhc3MKCiAgICBjbGFzcyBDb3VudEFmdGVyOgogICAgICAgIGRlZiBfX2luaXRfXygKICAgICAgICAgICAgc2VsZiwgbG9jaywgY291bnRfPU5vbmUsIGFmdGVyXz1Ob25lLCBhY3Rpb25fPU5vbmUsIGFjdGlvbl9hcmdzXz1bXQogICAgICAgICk6CiAgICAgICAgICAgIHNlbGYubG9jayA9IGxvY2sKCiAgICAgICAgICAgIHNlbGYuY291bnQgPSBjb3VudF8gaWYgY291bnRfIGlzIG5vdCBOb25lIGVsc2UgSU5GX0NPVU5UCiAgICAgICAgICAgIHNlbGYuYWZ0ZXIgPSBhZnRlcl8gaWYgYWZ0ZXJfIGlzIG5vdCBOb25lIGVsc2UgMAogICAgICAgICAgICBzZWxmLmFjdGlvbiA9IGFjdGlvbl8KICAgICAgICAgICAgc2VsZi5hY3Rpb25fYXJncyA9IGFjdGlvbl9hcmdzXwoKICAgICAgICAgICAgaWYgc2VsZi5hY3Rpb24gPT0gImNvbm5lY3Rpb25fcmVmdXNlZCI6CiAgICAgICAgICAgICAgICBzZWxmLmVycm9yX2hhbmRsZXIgPSBfU2VydmVyUnVudGltZS5Db25uZWN0aW9uUmVmdXNlZEFjdGlvbigpCiAgICAgICAgICAgIGVsaWYgc2VsZi5hY3Rpb24gPT0gImNvbm5lY3Rpb25fcmVzZXRfYnlfcGVlciI6CiAgICAgICAgICAgICAgICBzZWxmLmVycm9yX2hhbmRsZXIgPSBfU2VydmVyUnVudGltZS5Db25uZWN0aW9uUmVzZXRCeVBlZXJBY3Rpb24oCiAgICAgICAgICAgICAgICAgICAgKnNlbGYuYWN0aW9uX2FyZ3MKICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgZWxpZiBzZWxmLmFjdGlvbiA9PSAiYnJva2VuX3BpcGUiOgogICAgICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyID0gX1NlcnZlclJ1bnRpbWUuQnJva2VuUGlwZUFjdGlvbigpCiAgICAgICAgICAgIGVsaWYgc2VsZi5hY3Rpb24gPT0gInJlZGlyZWN0X3RvIjoKICAgICAgICAgICAgICAgIHNlbGYuZXJyb3JfaGFuZGxlciA9IF9TZXJ2ZXJSdW50aW1lLlJlZGlyZWN0QWN0aW9uKCpzZWxmLmFjdGlvbl9hcmdzKQogICAgICAgICAgICBlbGlmIHNlbGYuYWN0aW9uID09ICJzbG93X2Rvd24iOgogICAgICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyID0gX1NlcnZlclJ1bnRpbWUuU2xvd0Rvd25BY3Rpb24oKnNlbGYuYWN0aW9uX2FyZ3MpCiAgICAgICAgICAgIGVsaWYgc2VsZi5hY3Rpb24gPT0gInFwc19saW1pdF9leGNlZWRlZCI6CiAgICAgICAgICAgICAgICBzZWxmLmVycm9yX2hhbmRsZXIgPSBfU2VydmVyUnVudGltZS5RcHNMaW1pdEV4Y2VlZGVkQWN0aW9uKAogICAgICAgICAgICAgICAgICAgICpzZWxmLmFjdGlvbl9hcmdzCiAgICAgICAgICAgICAgICApCiAgICAgICAgICAgIGVsaWYgc2VsZi5hY3Rpb24gPT0gInRvdGFsX3Fwc19saW1pdF9leGNlZWRlZCI6CiAgICAgICAgICAgICAgICBzZWxmLmVycm9yX2hhbmRsZXIgPSBfU2VydmVyUnVudGltZS5Ub3RhbFFwc0xpbWl0RXhjZWVkZWRBY3Rpb24oCiAgICAgICAgICAgICAgICAgICAgKnNlbGYuYWN0aW9uX2FyZ3MKICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgZWxzZToKICAgICAgICAgICAgICAgIHNlbGYuZXJyb3JfaGFuZGxlciA9IF9TZXJ2ZXJSdW50aW1lLkV4cGVjdGVkNTAwRXJyb3JBY3Rpb24oKQoKICAgICAgICBAc3RhdGljbWV0aG9kCiAgICAgICAgZGVmIGZyb21fY2dpX3BhcmFtcyhsb2NrLCBwYXJhbXMpOgogICAgICAgICAgICByZXR1cm4gX1NlcnZlclJ1bnRpbWUuQ291bnRBZnRlcigKICAgICAgICAgICAgICAgIGxvY2s9bG9jaywKICAgICAgICAgICAgICAgIGNvdW50Xz1fYW5kX3RoZW4ocGFyYW1zLmdldCgiY291bnQiLCBbTm9uZV0pWzBdLCBpbnQpLAogICAgICAgICAgICAgICAgYWZ0ZXJfPV9hbmRfdGhlbihwYXJhbXMuZ2V0KCJhZnRlciIsIFtOb25lXSlbMF0sIGludCksCiAgICAgICAgICAgICAgICBhY3Rpb25fPXBhcmFtcy5nZXQoImFjdGlvbiIsIFtOb25lXSlbMF0sCiAgICAgICAgICAgICAgICBhY3Rpb25fYXJnc189cGFyYW1zLmdldCgiYWN0aW9uX2FyZ3MiLCBbXSksCiAgICAgICAgICAgICkKCiAgICAgICAgZGVmIF9fc3RyX18oc2VsZik6CiAgICAgICAgICAgIHJldHVybiBmImNvdW50OntzZWxmLmNvdW50fSBhZnRlcjp7c2VsZi5hZnRlcn0gYWN0aW9uOntzZWxmLmFjdGlvbn0gYWN0aW9uX2FyZ3M6e3NlbGYuYWN0aW9uX2FyZ3N9IgoKICAgICAgICBkZWYgaGFzX2VmZmVjdChzZWxmKToKICAgICAgICAgICAgd2l0aCBzZWxmLmxvY2s6CiAgICAgICAgICAgICAgICBpZiBzZWxmLmFmdGVyOgogICAgICAgICAgICAgICAgICAgIHNlbGYuYWZ0ZXIgLT0gMQogICAgICAgICAgICAgICAgaWYgc2VsZi5hZnRlciA9PSAwOgogICAgICAgICAgICAgICAgICAgIGlmIHNlbGYuY291bnQ6CiAgICAgICAgICAgICAgICAgICAgICAgIHNlbGYuY291bnQgLT0gMQogICAgICAgICAgICAgICAgICAgICAgICByZXR1cm4gVHJ1ZQogICAgICAgICAgICAgICAgcmV0dXJuIEZhbHNlCgogICAgICAgIGRlZiBpbmplY3RfZXJyb3Ioc2VsZiwgcmVxdWVzdF9oYW5kbGVyKToKICAgICAgICAgICAgc2VsZi5lcnJvcl9oYW5kbGVyLmluamVjdF9lcnJvcihyZXF1ZXN0X2hhbmRsZXIpCgogICAgZGVmIF9faW5pdF9fKHNlbGYpOgogICAgICAgIHNlbGYubG9jayA9IHRocmVhZGluZy5Mb2NrKCkKICAgICAgICBzZWxmLmF0X3BhcnRfdXBsb2FkID0gTm9uZQogICAgICAgIHNlbGYuYXRfb2JqZWN0X3VwbG9hZCA9IE5vbmUKICAgICAgICBzZWxmLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IE5vbmUKICAgICAgICBzZWxmLmZha2VfdXBsb2FkcyA9IGRpY3QoKQogICAgICAgIHNlbGYuc2xvd19wdXQgPSBOb25lCiAgICAgICAgc2VsZi5mYWtlX211bHRpcGFydF91cGxvYWQgPSBOb25lCiAgICAgICAgc2VsZi5hdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQgPSBOb25lCgogICAgZGVmIHJlZ2lzdGVyX2Zha2VfdXBsb2FkKHNlbGYsIHVwbG9hZF9pZCwga2V5KToKICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgc2VsZi5mYWtlX3VwbG9hZHNbdXBsb2FkX2lkXSA9IGtleQoKICAgIGRlZiBpc19mYWtlX3VwbG9hZChzZWxmLCB1cGxvYWRfaWQsIGtleSk6CiAgICAgICAgd2l0aCBzZWxmLmxvY2s6CiAgICAgICAgICAgIGlmIHVwbG9hZF9pZCBpbiBzZWxmLmZha2VfdXBsb2FkczoKICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLmZha2VfdXBsb2Fkc1t1cGxvYWRfaWRdID09IGtleQogICAgICAgIHJldHVybiBGYWxzZQoKICAgIGRlZiByZXNldChzZWxmKToKICAgICAgICB3aXRoIHNlbGYubG9jazoKICAgICAgICAgICAgc2VsZi5hdF9wYXJ0X3VwbG9hZCA9IE5vbmUKICAgICAgICAgICAgc2VsZi5hdF9vYmplY3RfdXBsb2FkID0gTm9uZQogICAgICAgICAgICBzZWxmLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciA9IE5vbmUKICAgICAgICAgICAgc2VsZi5mYWtlX3VwbG9hZHMgPSBkaWN0KCkKICAgICAgICAgICAgc2VsZi5zbG93X3B1dCA9IE5vbmUKICAgICAgICAgICAgc2VsZi5mYWtlX211bHRpcGFydF91cGxvYWQgPSBOb25lCiAgICAgICAgICAgIHNlbGYuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkID0gTm9uZQoKCl9ydW50aW1lID0gX1NlcnZlclJ1bnRpbWUoKQoKCmRlZiBnZXRfcmFuZG9tX3N0cmluZyhsZW5ndGgpOgogICAgIyBjaG9vc2UgZnJvbSBhbGwgbG93ZXJjYXNlIGxldHRlcgogICAgbGV0dGVycyA9IHN0cmluZy5hc2NpaV9sb3dlcmNhc2UKICAgIHJlc3VsdF9zdHIgPSAiIi5qb2luKHJhbmRvbS5jaG9pY2UobGV0dGVycykgZm9yIGkgaW4gcmFuZ2UobGVuZ3RoKSkKICAgIHJldHVybiByZXN1bHRfc3RyCgoKY2xhc3MgUmVxdWVzdEhhbmRsZXIoaHR0cC5zZXJ2ZXIuQmFzZUhUVFBSZXF1ZXN0SGFuZGxlcik6CiAgICBkZWYgX29rKHNlbGYpOgogICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQvcGxhaW4iKQogICAgICAgIHNlbGYuZW5kX2hlYWRlcnMoKQogICAgICAgIHNlbGYud2ZpbGUud3JpdGUoYiJPSyIpCgogICAgZGVmIF9waW5nKHNlbGYpOgogICAgICAgIHNlbGYuX29rKCkKCiAgICBkZWYgcmVhZF9hbGxfaW5wdXQoc2VsZik6CiAgICAgICAgY29udGVudF9sZW5ndGggPSBpbnQoc2VsZi5oZWFkZXJzLmdldCgiQ29udGVudC1MZW5ndGgiLCAwKSkKICAgICAgICB0b19yZWFkID0gY29udGVudF9sZW5ndGgKICAgICAgICB3aGlsZSB0b19yZWFkID4gMDoKICAgICAgICAgICAgIyByZWFkIGNvbnRlbnQgaW4gb3JkZXIgdG8gYXZvaWQgZXJyb3Igb24gY2xpZW50CiAgICAgICAgICAgICMgUG9jbzo6RXhjZXB0aW9uLiBDb2RlOiAxMDAwLCBlLmNvZGUoKSA9IDMyLCBJL08gZXJyb3I6IEJyb2tlbiBwaXBlCiAgICAgICAgICAgICMgZG8gaXQgcGllY2UgYnkgcGllY2UgaW4gb3JkZXIgdG8gYXZvaWQgYmlnIGFsbG9jYXRpb24KICAgICAgICAgICAgc2l6ZSA9IG1pbih0b19yZWFkLCAxMDI0KQogICAgICAgICAgICBzdHIoc2VsZi5yZmlsZS5yZWFkKHNpemUpKQogICAgICAgICAgICB0b19yZWFkIC09IHNpemUKCiAgICBkZWYgcmVkaXJlY3Qoc2VsZiwgaG9zdD1Ob25lLCBwb3J0PU5vbmUpOgogICAgICAgIGlmIGhvc3QgaXMgTm9uZSBhbmQgcG9ydCBpcyBOb25lOgogICAgICAgICAgICBob3N0ID0gc2VsZi5zZXJ2ZXIudXBzdHJlYW1faG9zdAogICAgICAgICAgICBwb3J0ID0gc2VsZi5zZXJ2ZXIudXBzdHJlYW1fcG9ydAoKICAgICAgICBzZWxmLnJlYWRfYWxsX2lucHV0KCkKCiAgICAgICAgc2VsZi5zZW5kX3Jlc3BvbnNlKDMwNykKICAgICAgICB1cmwgPSBmImh0dHA6Ly97aG9zdH06e3BvcnR9e3NlbGYucGF0aH0iCiAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgicmVkaXJlY3QgdG8gJXMiLCB1cmwpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiTG9jYXRpb24iLCB1cmwpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCiAgICAgICAgc2VsZi53ZmlsZS53cml0ZShiIlJlZGlyZWN0ZWQiKQoKICAgIGRlZiB3cml0ZV9lcnJvcihzZWxmLCBodHRwX2NvZGUsIGRhdGEsIGNvbnRlbnRfbGVuZ3RoPU5vbmUpOgogICAgICAgIGlmIGNvbnRlbnRfbGVuZ3RoIGlzIE5vbmU6CiAgICAgICAgICAgIGNvbnRlbnRfbGVuZ3RoID0gbGVuKGRhdGEpCiAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgid3JpdGVfZXJyb3IgJXMiLCBkYXRhKQogICAgICAgIHNlbGYucmVhZF9hbGxfaW5wdXQoKQogICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZShodHRwX2NvZGUpCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQveG1sIikKICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIHN0cihjb250ZW50X2xlbmd0aCkpCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCiAgICAgICAgaWYgZGF0YToKICAgICAgICAgICAgc2VsZi53ZmlsZS53cml0ZShieXRlcyhkYXRhLCAiVVRGLTgiKSkKCiAgICBkZWYgX2Zha2VfcHV0X29rKHNlbGYpOgogICAgICAgIHNlbGYubG9nX21lc3NhZ2UoImZha2UgcHV0IikKCiAgICAgICAgc2VsZi5yZWFkX2FsbF9pbnB1dCgpCgogICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQveG1sIikKICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJFVGFnIiwgImI1NDM1N2ZhZjA2MzJjY2U0NmU5NDJmYTY4MzU2YjM4IikKICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIDApCiAgICAgICAgc2VsZi5lbmRfaGVhZGVycygpCgogICAgZGVmIF9mYWtlX3VwbG9hZHMoc2VsZiwgcGF0aCwgdXBsb2FkX2lkKToKICAgICAgICBzZWxmLnJlYWRfYWxsX2lucHV0KCkKCiAgICAgICAgcGFydHMgPSBbeCBmb3IgeCBpbiBwYXRoLnNwbGl0KCIvIikgaWYgeF0KICAgICAgICBidWNrZXQgPSBwYXJ0c1swXQogICAgICAgIGtleSA9ICIvIi5qb2luKHBhcnRzWzE6XSkKICAgICAgICBkYXRhID0gKAogICAgICAgICAgICAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz5cbicKICAgICAgICAgICAgIjxJbml0aWF0ZU11bHRpcGFydFVwbG9hZFJlc3VsdD5cbiIKICAgICAgICAgICAgZiI8QnVja2V0PntidWNrZXR9PC9CdWNrZXQ+IgogICAgICAgICAgICBmIjxLZXk+e2tleX08L0tleT4iCiAgICAgICAgICAgIGYiPFVwbG9hZElkPnt1cGxvYWRfaWR9PC9VcGxvYWRJZD4iCiAgICAgICAgICAgICI8L0luaXRpYXRlTXVsdGlwYXJ0VXBsb2FkUmVzdWx0PiIKICAgICAgICApCgogICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQveG1sIikKICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIGxlbihkYXRhKSkKICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKCiAgICAgICAgc2VsZi53ZmlsZS53cml0ZShieXRlcyhkYXRhLCAiVVRGLTgiKSkKCiAgICBkZWYgX2Zha2VfcG9zdF9vayhzZWxmLCBwYXRoKToKICAgICAgICBzZWxmLnJlYWRfYWxsX2lucHV0KCkKCiAgICAgICAgcGFydHMgPSBbeCBmb3IgeCBpbiBwYXRoLnNwbGl0KCIvIikgaWYgeF0KICAgICAgICBidWNrZXQgPSBwYXJ0c1swXQogICAgICAgIGtleSA9ICIvIi5qb2luKHBhcnRzWzE6XSkKICAgICAgICBsb2NhdGlvbiA9ICJodHRwOi8vRXhhbXBsZS1CdWNrZXQuczMuUmVnaW9uLmFtYXpvbmF3cy5jb20vIiArIHBhdGgKICAgICAgICBkYXRhID0gKAogICAgICAgICAgICAnPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz5cbicKICAgICAgICAgICAgIjxDb21wbGV0ZU11bHRpcGFydFVwbG9hZFJlc3VsdD5cbiIKICAgICAgICAgICAgZiI8TG9jYXRpb24+e2xvY2F0aW9ufTwvTG9jYXRpb24+XG4iCiAgICAgICAgICAgIGYiPEJ1Y2tldD57YnVja2V0fTwvQnVja2V0PlxuIgogICAgICAgICAgICBmIjxLZXk+e2tleX08L0tleT5cbiIKICAgICAgICAgICAgZic8RVRhZz4iMzg1OGY2MjIzMGFjM2M5MTVmMzAwYzY2NDMxMmMxMWYtOSI8L0VUYWc+XG4nCiAgICAgICAgICAgIGYiPC9Db21wbGV0ZU11bHRpcGFydFVwbG9hZFJlc3VsdD5cbiIKICAgICAgICApCgogICAgICAgIHNlbGYuc2VuZF9yZXNwb25zZSgyMDApCiAgICAgICAgc2VsZi5zZW5kX2hlYWRlcigiQ29udGVudC1UeXBlIiwgInRleHQveG1sIikKICAgICAgICBzZWxmLnNlbmRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIGxlbihkYXRhKSkKICAgICAgICBzZWxmLmVuZF9oZWFkZXJzKCkKCiAgICAgICAgc2VsZi53ZmlsZS53cml0ZShieXRlcyhkYXRhLCAiVVRGLTgiKSkKCiAgICBkZWYgX21vY2tfc2V0dGluZ3Moc2VsZik6CiAgICAgICAgcGFydHMgPSB1cmxsaWIucGFyc2UudXJsc3BsaXQoc2VsZi5wYXRoKQogICAgICAgIHBhdGggPSBbeCBmb3IgeCBpbiBwYXJ0cy5wYXRoLnNwbGl0KCIvIikgaWYgeF0KICAgICAgICBhc3NlcnQgcGF0aFswXSA9PSAibW9ja19zZXR0aW5ncyIsIHBhdGgKICAgICAgICBpZiBsZW4ocGF0aCkgPCAyOgogICAgICAgICAgICByZXR1cm4gc2VsZi53cml0ZV9lcnJvcig0MDAsICJfbW9ja19zZXR0aW5nczogd3JvbmcgY29tbWFuZCIpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gImF0X3BhcnRfdXBsb2FkIjoKICAgICAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICAgICAgX3J1bnRpbWUuYXRfcGFydF91cGxvYWQgPSBfU2VydmVyUnVudGltZS5Db3VudEFmdGVyLmZyb21fY2dpX3BhcmFtcygKICAgICAgICAgICAgICAgIF9ydW50aW1lLmxvY2ssIHBhcmFtcwogICAgICAgICAgICApCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoInNldCBhdF9wYXJ0X3VwbG9hZCAlcyIsIF9ydW50aW1lLmF0X3BhcnRfdXBsb2FkKQogICAgICAgICAgICByZXR1cm4gc2VsZi5fb2soKQoKICAgICAgICBpZiBwYXRoWzFdID09ICJhdF9vYmplY3RfdXBsb2FkIjoKICAgICAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICAgICAgX3J1bnRpbWUuYXRfb2JqZWN0X3VwbG9hZCA9IF9TZXJ2ZXJSdW50aW1lLkNvdW50QWZ0ZXIuZnJvbV9jZ2lfcGFyYW1zKAogICAgICAgICAgICAgICAgX3J1bnRpbWUubG9jaywgcGFyYW1zCiAgICAgICAgICAgICkKICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2V0IGF0X29iamVjdF91cGxvYWQgJXMiLCBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkKQogICAgICAgICAgICByZXR1cm4gc2VsZi5fb2soKQoKICAgICAgICBpZiBwYXRoWzFdID09ICJmYWtlX3B1dHMiOgogICAgICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPUZhbHNlKQogICAgICAgICAgICBfcnVudGltZS5mYWtlX3B1dF93aGVuX2xlbmd0aF9iaWdnZXIgPSBpbnQoCiAgICAgICAgICAgICAgICBwYXJhbXMuZ2V0KCJ3aGVuX2xlbmd0aF9iaWdnZXIiLCBbMTAyNCAqIDEwMjRdKVswXQogICAgICAgICAgICApCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoInNldCBmYWtlX3B1dHMgJXMiLCBfcnVudGltZS5mYWtlX3B1dF93aGVuX2xlbmd0aF9iaWdnZXIpCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gInNsb3dfcHV0IjoKICAgICAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICAgICAgX3J1bnRpbWUuc2xvd19wdXQgPSBfU2VydmVyUnVudGltZS5TbG93UHV0KAogICAgICAgICAgICAgICAgbG9jaz1fcnVudGltZS5sb2NrLAogICAgICAgICAgICAgICAgbWluaW1hbF9sZW5ndGhfPV9hbmRfdGhlbihwYXJhbXMuZ2V0KCJtaW5pbWFsX2xlbmd0aCIsIFtOb25lXSlbMF0sIGludCksCiAgICAgICAgICAgICAgICBwcm9iYWJpbGl0eV89X2FuZF90aGVuKHBhcmFtcy5nZXQoInByb2JhYmlsaXR5IiwgW05vbmVdKVswXSwgZmxvYXQpLAogICAgICAgICAgICAgICAgdGltZW91dF89X2FuZF90aGVuKHBhcmFtcy5nZXQoInRpbWVvdXQiLCBbTm9uZV0pWzBdLCBmbG9hdCksCiAgICAgICAgICAgICAgICBjb3VudF89X2FuZF90aGVuKHBhcmFtcy5nZXQoImNvdW50IiwgW05vbmVdKVswXSwgaW50KSwKICAgICAgICAgICAgKQogICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKCJzZXQgc2xvdyBwdXQgJXMiLCBfcnVudGltZS5zbG93X3B1dCkKICAgICAgICAgICAgcmV0dXJuIHNlbGYuX29rKCkKCiAgICAgICAgaWYgcGF0aFsxXSA9PSAic2V0dXBfZmFrZV9tdWx0cGFydHVwbG9hZHMiOgogICAgICAgICAgICBfcnVudGltZS5mYWtlX211bHRpcGFydF91cGxvYWQgPSBUcnVlCiAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoInNldCBzZXR1cF9mYWtlX211bHRwYXJ0dXBsb2FkcyIpCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gImF0X2NyZWF0ZV9tdWx0aV9wYXJ0X3VwbG9hZCI6CiAgICAgICAgICAgIHBhcmFtcyA9IHVybGxpYi5wYXJzZS5wYXJzZV9xcyhwYXJ0cy5xdWVyeSwga2VlcF9ibGFua192YWx1ZXM9RmFsc2UpCiAgICAgICAgICAgIF9ydW50aW1lLmF0X2NyZWF0ZV9tdWx0aV9wYXJ0X3VwbG9hZCA9ICgKICAgICAgICAgICAgICAgIF9TZXJ2ZXJSdW50aW1lLkNvdW50QWZ0ZXIuZnJvbV9jZ2lfcGFyYW1zKF9ydW50aW1lLmxvY2ssIHBhcmFtcykKICAgICAgICAgICAgKQogICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKAogICAgICAgICAgICAgICAgInNldCBhdF9jcmVhdGVfbXVsdGlfcGFydF91cGxvYWQgJXMiLAogICAgICAgICAgICAgICAgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkLAogICAgICAgICAgICApCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIGlmIHBhdGhbMV0gPT0gInJlc2V0IjoKICAgICAgICAgICAgX3J1bnRpbWUucmVzZXQoKQogICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKCJyZXNldCIpCiAgICAgICAgICAgIHJldHVybiBzZWxmLl9vaygpCgogICAgICAgIHJldHVybiBzZWxmLndyaXRlX2Vycm9yKDQwMCwgIl9tb2NrX3NldHRpbmdzOiB3cm9uZyBjb21tYW5kIikKCiAgICBkZWYgZG9fR0VUKHNlbGYpOgogICAgICAgIGlmIHNlbGYucGF0aCA9PSAiLyI6CiAgICAgICAgICAgIHJldHVybiBzZWxmLl9waW5nKCkKCiAgICAgICAgaWYgc2VsZi5wYXRoLnN0YXJ0c3dpdGgoIi9tb2NrX3NldHRpbmdzIik6CiAgICAgICAgICAgIHJldHVybiBzZWxmLl9tb2NrX3NldHRpbmdzKCkKCiAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgiZ2V0IHJlZGlyZWN0IikKICAgICAgICByZXR1cm4gc2VsZi5yZWRpcmVjdCgpCgogICAgZGVmIGRvX1BVVChzZWxmKToKICAgICAgICBjb250ZW50X2xlbmd0aCA9IGludChzZWxmLmhlYWRlcnMuZ2V0KCJDb250ZW50LUxlbmd0aCIsIDApKQoKICAgICAgICBpZiBfcnVudGltZS5zbG93X3B1dCBpcyBub3QgTm9uZToKICAgICAgICAgICAgdGltZW91dCA9IF9ydW50aW1lLnNsb3dfcHV0LmdldF90aW1lb3V0KGNvbnRlbnRfbGVuZ3RoKQogICAgICAgICAgICBpZiB0aW1lb3V0IGlzIG5vdCBOb25lOgogICAgICAgICAgICAgICAgc2VsZi5sb2dfbWVzc2FnZSgic2xvdyBwdXQgJXMiLCB0aW1lb3V0KQogICAgICAgICAgICAgICAgdGltZS5zbGVlcCh0aW1lb3V0KQoKICAgICAgICBwYXJ0cyA9IHVybGxpYi5wYXJzZS51cmxzcGxpdChzZWxmLnBhdGgpCiAgICAgICAgcGFyYW1zID0gdXJsbGliLnBhcnNlLnBhcnNlX3FzKHBhcnRzLnF1ZXJ5LCBrZWVwX2JsYW5rX3ZhbHVlcz1GYWxzZSkKICAgICAgICB1cGxvYWRfaWQgPSBwYXJhbXMuZ2V0KCJ1cGxvYWRJZCIsIFtOb25lXSlbMF0KCiAgICAgICAgaWYgdXBsb2FkX2lkIGlzIG5vdCBOb25lOgogICAgICAgICAgICBpZiBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCBpcyBub3QgTm9uZToKICAgICAgICAgICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICAgICAgICAgInB1dCBhdF9wYXJ0X3VwbG9hZCAlcywgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZCwKICAgICAgICAgICAgICAgICAgICB1cGxvYWRfaWQsCiAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICApCgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfcGFydF91cGxvYWQuaGFzX2VmZmVjdCgpOgogICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5hdF9wYXJ0X3VwbG9hZC5pbmplY3RfZXJyb3Ioc2VsZikKICAgICAgICAgICAgaWYgX3J1bnRpbWUuZmFrZV9tdWx0aXBhcnRfdXBsb2FkOgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuaXNfZmFrZV91cGxvYWQodXBsb2FkX2lkLCBwYXJ0cy5wYXRoKToKICAgICAgICAgICAgICAgICAgICByZXR1cm4gc2VsZi5fZmFrZV9wdXRfb2soKQogICAgICAgIGVsc2U6CiAgICAgICAgICAgIGlmIF9ydW50aW1lLmF0X29iamVjdF91cGxvYWQgaXMgbm90IE5vbmU6CiAgICAgICAgICAgICAgICBpZiBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkLmhhc19lZmZlY3QoKToKICAgICAgICAgICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKAogICAgICAgICAgICAgICAgICAgICAgICAicHV0IGVycm9yX2F0X29iamVjdF91cGxvYWQgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuYXRfb2JqZWN0X3VwbG9hZCwKICAgICAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgICAgIHJldHVybiBfcnVudGltZS5hdF9vYmplY3RfdXBsb2FkLmluamVjdF9lcnJvcihzZWxmKQogICAgICAgICAgICBpZiBfcnVudGltZS5mYWtlX3B1dF93aGVuX2xlbmd0aF9iaWdnZXIgaXMgbm90IE5vbmU6CiAgICAgICAgICAgICAgICBpZiBjb250ZW50X2xlbmd0aCA+IF9ydW50aW1lLmZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlcjoKICAgICAgICAgICAgICAgICAgICBzZWxmLmxvZ19tZXNzYWdlKAogICAgICAgICAgICAgICAgICAgICAgICAicHV0IGZha2VfcHV0X3doZW5fbGVuZ3RoX2JpZ2dlciAlcywgJXMsICVzIiwKICAgICAgICAgICAgICAgICAgICAgICAgX3J1bnRpbWUuZmFrZV9wdXRfd2hlbl9sZW5ndGhfYmlnZ2VyLAogICAgICAgICAgICAgICAgICAgICAgICBjb250ZW50X2xlbmd0aCwKICAgICAgICAgICAgICAgICAgICAgICAgcGFydHMsCiAgICAgICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLl9mYWtlX3B1dF9vaygpCgogICAgICAgIHNlbGYubG9nX21lc3NhZ2UoCiAgICAgICAgICAgICJwdXQgcmVkaXJlY3QgJXMiLAogICAgICAgICAgICBwYXJ0cywKICAgICAgICApCiAgICAgICAgcmV0dXJuIHNlbGYucmVkaXJlY3QoKQoKICAgIGRlZiBkb19QT1NUKHNlbGYpOgogICAgICAgIHBhcnRzID0gdXJsbGliLnBhcnNlLnVybHNwbGl0KHNlbGYucGF0aCkKICAgICAgICBwYXJhbXMgPSB1cmxsaWIucGFyc2UucGFyc2VfcXMocGFydHMucXVlcnksIGtlZXBfYmxhbmtfdmFsdWVzPVRydWUpCiAgICAgICAgdXBsb2FkcyA9IHBhcmFtcy5nZXQoInVwbG9hZHMiLCBbTm9uZV0pWzBdCiAgICAgICAgaWYgdXBsb2FkcyBpcyBub3QgTm9uZToKICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkIGlzIG5vdCBOb25lOgogICAgICAgICAgICAgICAgaWYgX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkLmhhc19lZmZlY3QoKToKICAgICAgICAgICAgICAgICAgICByZXR1cm4gX3J1bnRpbWUuYXRfY3JlYXRlX211bHRpX3BhcnRfdXBsb2FkLmluamVjdF9lcnJvcihzZWxmKQoKICAgICAgICAgICAgaWYgX3J1bnRpbWUuZmFrZV9tdWx0aXBhcnRfdXBsb2FkOgogICAgICAgICAgICAgICAgdXBsb2FkX2lkID0gZ2V0X3JhbmRvbV9zdHJpbmcoNSkKICAgICAgICAgICAgICAgIF9ydW50aW1lLnJlZ2lzdGVyX2Zha2VfdXBsb2FkKHVwbG9hZF9pZCwgcGFydHMucGF0aCkKICAgICAgICAgICAgICAgIHJldHVybiBzZWxmLl9mYWtlX3VwbG9hZHMocGFydHMucGF0aCwgdXBsb2FkX2lkKQoKICAgICAgICB1cGxvYWRfaWQgPSBwYXJhbXMuZ2V0KCJ1cGxvYWRJZCIsIFtOb25lXSlbMF0KICAgICAgICBpZiBfcnVudGltZS5pc19mYWtlX3VwbG9hZCh1cGxvYWRfaWQsIHBhcnRzLnBhdGgpOgogICAgICAgICAgICByZXR1cm4gc2VsZi5fZmFrZV9wb3N0X29rKHBhcnRzLnBhdGgpCgogICAgICAgIHJldHVybiBzZWxmLnJlZGlyZWN0KCkKCiAgICBkZWYgZG9fSEVBRChzZWxmKToKICAgICAgICBzZWxmLnJlZGlyZWN0KCkKCiAgICBkZWYgZG9fREVMRVRFKHNlbGYpOgogICAgICAgIHNlbGYucmVkaXJlY3QoKQoKCmNsYXNzIF9UaHJlYWRlZEhUVFBTZXJ2ZXIoc29ja2V0c2VydmVyLlRocmVhZGluZ01peEluLCBodHRwLnNlcnZlci5IVFRQU2VydmVyKToKICAgICIiIkhhbmRsZSByZXF1ZXN0cyBpbiBhIHNlcGFyYXRlIHRocmVhZC4iIiIKCiAgICBkZWYgc2V0X3Vwc3RyZWFtKHNlbGYsIHVwc3RyZWFtX2hvc3QsIHVwc3RyZWFtX3BvcnQpOgogICAgICAgIHNlbGYudXBzdHJlYW1faG9zdCA9IHVwc3RyZWFtX2hvc3QKICAgICAgICBzZWxmLnVwc3RyZWFtX3BvcnQgPSB1cHN0cmVhbV9wb3J0CgoKaWYgX19uYW1lX18gPT0gIl9fbWFpbl9fIjoKICAgIGh0dHBkID0gX1RocmVhZGVkSFRUUFNlcnZlcigoIjAuMC4wLjAiLCBpbnQoc3lzLmFyZ3ZbMV0pKSwgUmVxdWVzdEhhbmRsZXIpCiAgICBpZiBsZW4oc3lzLmFyZ3YpID09IDQ6CiAgICAgICAgaHR0cGQuc2V0X3Vwc3RyZWFtKHN5cy5hcmd2WzJdLCBzeXMuYXJndlszXSkKICAgIGVsc2U6CiAgICAgICAgaHR0cGQuc2V0X3Vwc3RyZWFtKCJtaW5pbzEiLCA5MDAxKQogICAgaHR0cGQuc2VydmVfZm9yZXZlcigpCg== | base64 --decode > broken_s3.py] Stdout:2457 run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 detach:True nothrow:False cmd: ['bash', '-c', 'python3 broken_s3.py 8083 >/var/log/resolver/broken_s3.log 2>/var/log/resolver/broken_s3.err.log'] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 bash -c python3 broken_s3.py 8083 >/var/log/resolver/broken_s3.log 2>/var/log/resolver/broken_s3.err.log] run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/'] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 curl -s http://localhost:8083/] Executing query SELECT count(*) FROM test_named_colections on node1 Exitcode:7 Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/.env --project-name roottests3accessheaders-gw4 --file /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml stop --timeout 20] [gw4] PASSED test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header] Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:772 Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 Stderr: Container roottestprometheusendpoint-gw2-node-1 Stopping Stderr: Container roottestprometheusendpoint-gw2-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/.env --project-name roottestprometheusendpoint-gw2 --file /ClickHouse/tests/integration/test_prometheus_endpoint/_instances-0-gw2/node/docker-compose.yml down --volumes] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:2457 run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8083/'] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 curl -s http://localhost:8083/] Connecting to 172.16.5.3(172.16.5.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=2516, time_out=30000, session_id=8, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Stdout:OK broken_s3.py answered OK on attempt 2 Mock server broken_s3.py started run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 9 ? 00:00:00 clickhouse run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 bash -c pkill clickhouse] run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:9 Executing query SELECT name FROM system.parts where name = 'all_1_1_4' and table = 'table_for_recompression' on node2 run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'rm -r /var/lib/clickhouse/store/test_rocksdb_read_only_missing'] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c rm -r /var/lib/clickhouse/store/test_rocksdb_read_only_missing] Stderr: Container roottestprometheusendpoint-gw2-node-1 Stopping Stderr: Container roottestprometheusendpoint-gw2-node-1 Stopped Stderr: Container roottestprometheusendpoint-gw2-node-1 Removing Stderr: Container roottestprometheusendpoint-gw2-node-1 Removed Stderr: Network roottestprometheusendpoint-gw2_default Removing Stderr: Network roottestprometheusendpoint-gw2_default Removed Cleanup called Docker networks for project roottestprometheusendpoint-gw2 are NETWORK ID NAME DRIVER SCOPE run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Docker containers for project roottestprometheusendpoint-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestprometheusendpoint-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestprometheusendpoint-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrocksdbreadonly-gw3-node-1/exec HTTP/1.1" 201 74 Unstopped containers: {} No running containers for project: roottestprometheusendpoint-gw2 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] http://localhost:None "POST /v1.46/exec/14d80ab8ccf33674fe78e69bb95cbf108dfa7962b52dab349d1db6163741c4ea/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/14d80ab8ccf33674fe78e69bb95cbf108dfa7962b52dab349d1db6163741c4ea/json HTTP/1.1" 200 586 Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:7 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 7 Running tests in /ClickHouse/tests/integration/test_range_hashed_dictionary_types/test.py test_range_hashed_dictionary_types/test.py::test_range_hashed_dict Cluster start called. is_up=False Docker networks for project roottestrangehasheddictionarytypes-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrangehasheddictionarytypes-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrangehasheddictionarytypes-gw2 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestrangehasheddictionarytypes-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrangehasheddictionarytypes-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrangehasheddictionarytypes-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrangehasheddictionarytypes-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestrangehasheddictionarytypes-gw2 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:7 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 7 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/node1/database Setup logs dir /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/.env --project-name roottestrangehasheddictionarytypes-gw2 --file /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/node1/docker-compose.yml pull] Executing query OPTIMIZE TABLE table_for_recompression FINAL on node2 Executing query SELECT default_compression_codec FROM system.parts where name = 'all_1_1_4' on node2 Executing query SELECT recompression_ttl_info.expression FROM system.parts where name = 'all_1_1_4' on node2 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:2457 [gw6] PASSED test_recompression_ttl/test.py::test_recompression_multiple_ttls test_recompression_ttl/test.py::test_recompression_replicated Executing query CREATE TABLE recompression_replicated (d DateTime, key UInt64, data String) ENGINE ReplicatedMergeTree('/test/rr', '1') ORDER BY tuple() TTL d + INTERVAL 10 SECOND RECOMPRESS CODEC(ZSTD(13)) SETTINGS merge_with_recompression_ttl_timeout = 0 on node1 Executing query CREATE TABLE recompression_replicated (d DateTime, key UInt64, data String) ENGINE ReplicatedMergeTree('/test/rr', '2') ORDER BY tuple() TTL d + INTERVAL 10 SECOND RECOMPRESS CODEC(ZSTD(13)) SETTINGS merge_with_recompression_ttl_timeout = 0 on node2 Executing query INSERT INTO recompression_replicated VALUES (now(), 1, '1') on node1 run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:9 Stderr: Container roottestrole-gw0-instance-1 Stopping Stderr: Container roottestrole-gw0-instance-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_role/_instances-0-gw0/instance/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_role/_instances-0-gw0/instance/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_role/_instances-0-gw0/.env --project-name roottestrole-gw0 --file /ClickHouse/tests/integration/test_role/_instances-0-gw0/instance/docker-compose.yml down --volumes] Executing query SYSTEM SYNC REPLICA recompression_replicated on node2 Executing query SELECT default_compression_codec FROM system.parts where name = 'all_0_0_0' and table = 'recompression_replicated' on node1 run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:1593 Clickhouse process running. run container_id:roottestrocksdbreadonly-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrocksdbreadonly-gw3-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:1593 Executing query select 20 on node Executing query SELECT default_compression_codec FROM system.parts where name = 'all_0_0_0' and table = 'recompression_replicated' on node2 Executing query INSERT INTO test (key, value) VALUES (1, 'b'); on node Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Executing query SELECT * FROM test; on node Executing query DROP TABLE test; on node [gw3] PASSED test_rocksdb_read_only/test.py::test_dirctory_missing_after_stop test_rocksdb_read_only/test.py::test_read_only Executing query CREATE TABLE test (key UInt64, value String) Engine=EmbeddedRocksDB(0, '/var/lib/clickhouse/store/test_rocksdb_read_only', 1) PRIMARY KEY(key); on node Stderr: node1 Pulling Stderr: node1 Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/.env --project-name roottestrangehasheddictionarytypes-gw2 --file /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/node1/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/.env --project-name roottestrangehasheddictionarytypes-gw2 --file /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/node1/docker-compose.yml up -d --no-recreate] Executing query CREATE TABLE test (key UInt64, value String) Engine=EmbeddedRocksDB(0, '/var/lib/clickhouse/store/test_rocksdb_read_only') PRIMARY KEY(key); INSERT INTO test (key, value) VALUES (0, 'a'), (1, 'b'), (2, 'c'); on node Executing query CREATE TABLE test_fail (key UInt64, value String) Engine=EmbeddedRocksDB(0, '/var/lib/clickhouse/store/test_rocksdb_read_only') PRIMARY KEY(key); on node Stderr: Container roottestrole-gw0-instance-1 Stopping Stderr: Container roottestrole-gw0-instance-1 Stopped Stderr: Container roottestrole-gw0-instance-1 Removing Stderr: Container roottestrole-gw0-instance-1 Removed Stderr: Network roottestrole-gw0_default Removing Stderr: Network roottestrole-gw0_default Removed Cleanup called Docker networks for project roottestrole-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrole-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrole-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrole-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Stderr: node Pulling Stderr: node Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/.env --project-name roottestrestartserver-gw9 --file /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/.env --project-name roottestrestartserver-gw9 --file /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/node/docker-compose.yml up -d --no-recreate] Executing query CREATE TABLE test_fail (key UInt64, value String) Engine=EmbeddedRocksDB(10, '/var/lib/clickhouse/store/test_rocksdb_read_only') PRIMARY KEY(key); on node Unstopped containers: {} No running containers for project: roottestrole-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query CREATE TABLE test_1 (key UInt64, value String) Engine=EmbeddedRocksDB(0, '/var/lib/clickhouse/store/test_rocksdb_read_only', 1) PRIMARY KEY(key); DROP TABLE test_1; on node Stdout:7 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 7 test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order Running tests in /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/test.py Cluster start called. is_up=False Docker networks for project roottestprofilesettingsandconstraintsorder-gw0 are NETWORK ID NAME DRIVER SCOPE Stdout:2457 Docker containers for project roottestprofilesettingsandconstraintsorder-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestprofilesettingsandconstraintsorder-gw0 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestprofilesettingsandconstraintsorder-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestprofilesettingsandconstraintsorder-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestprofilesettingsandconstraintsorder-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestprofilesettingsandconstraintsorder-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Unstopped containers: {} No running containers for project: roottestprofilesettingsandconstraintsorder-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query CREATE TABLE test_2 (key UInt64, value String) Engine=EmbeddedRocksDB(10, '/var/lib/clickhouse/store/test_rocksdb_read_only', 1) PRIMARY KEY(key); DROP TABLE test_2; on node Stdout:7 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 7 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node1/database Setup logs dir /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node2/database Setup logs dir /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/.env --project-name roottestprofilesettingsandconstraintsorder-gw0 --file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node2/docker-compose.yml pull] Stdout:9 Executing query DROP TABLE test; CREATE TABLE test (key UInt64, value String) Engine=EmbeddedRocksDB(10, '/var/lib/clickhouse/store/test_rocksdb_read_only', 1) PRIMARY KEY(key); on node Stderr: Network roottestrangehasheddictionarytypes-gw2_default Creating Stderr: Network roottestrangehasheddictionarytypes-gw2_default Created Stderr: Container roottestrangehasheddictionarytypes-gw2-node1-1 Creating Stderr: Container roottestrangehasheddictionarytypes-gw2-node1-1 Created Stderr: Container roottestrangehasheddictionarytypes-gw2-node1-1 Starting Stderr: Container roottestrangehasheddictionarytypes-gw2-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestrangehasheddictionarytypes-gw2-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestrangehasheddictionarytypes-gw2-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.1.2... http://localhost:None "GET /v1.46/containers/roottestrangehasheddictionarytypes-gw2-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ad920f30b2ccbc02a68dd4ada77796c790465ed1a0e1a824f19db223095364fe/json HTTP/1.1" 200 None Executing query SELECT count() FROM test; on node http://localhost:None "GET /v1.46/containers/ad920f30b2ccbc02a68dd4ada77796c790465ed1a0e1a824f19db223095364fe/json HTTP/1.1" 200 None Executing query INSERT INTO test (key, value) VALUES (4, 'd'); on node http://localhost:None "GET /v1.46/containers/ad920f30b2ccbc02a68dd4ada77796c790465ed1a0e1a824f19db223095364fe/json HTTP/1.1" 200 None Stderr: Network roottestrestartserver-gw9_default Creating Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Stderr: Network roottestrestartserver-gw9_default Created Stderr: Container roottestrestartserver-gw9-node-1 Creating Stderr: Container roottestrestartserver-gw9-node-1 Created Stderr: Container roottestrestartserver-gw9-node-1 Starting Stderr: Container roottestrestartserver-gw9-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestrestartserver-gw9-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestrestartserver-gw9-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.3.2... Executing query DROP TABLE test; on node http://localhost:None "GET /v1.46/containers/roottestrestartserver-gw9-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/8765fd477c482971969c7dca95a6a88d2c4f74aeeaa40dbfb6b3537e79cb51fa/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ad920f30b2ccbc02a68dd4ada77796c790465ed1a0e1a824f19db223095364fe/json HTTP/1.1" 200 None Command:[docker compose --env-file /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/.env --project-name roottestrocksdbreadonly-gw3 --file /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/node/docker-compose.yml stop --timeout 20] [gw3] PASSED test_rocksdb_read_only/test.py::test_read_only http://localhost:None "GET /v1.46/containers/8765fd477c482971969c7dca95a6a88d2c4f74aeeaa40dbfb6b3537e79cb51fa/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ad920f30b2ccbc02a68dd4ada77796c790465ed1a0e1a824f19db223095364fe/json HTTP/1.1" 200 None ClickHouse node1 started run container_id:roottestrangehasheddictionarytypes-gw2-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "echo '4990954156238030839\t2018-12-31 21:00:00\t2020-12-30 20:59:59\t0.1\tRU' > /var/lib/clickhouse/user_files/rates.tsv"] Command:[docker exec roottestrangehasheddictionarytypes-gw2-node1-1 bash -c echo '4990954156238030839 2018-12-31 21:00:00 2020-12-30 20:59:59 0.1 RU' > /var/lib/clickhouse/user_files/rates.tsv] Executing query CREATE DICTIONARY rates ( hash_id UInt64, start_date DateTime default '0000-00-00 00:00:00', end_date DateTime default '0000-00-00 00:00:00', price Float64, currency String ) PRIMARY KEY hash_id SOURCE(file( path '/var/lib/clickhouse/user_files/rates.tsv' format 'TSV' )) LAYOUT(RANGE_HASHED()) RANGE(MIN start_date MAX end_date) LIFETIME(60); on node1 http://localhost:None "GET /v1.46/containers/8765fd477c482971969c7dca95a6a88d2c4f74aeeaa40dbfb6b3537e79cb51fa/json HTTP/1.1" 200 None Executing query SYSTEM RELOAD DICTIONARY default.rates on node1 http://localhost:None "GET /v1.46/containers/8765fd477c482971969c7dca95a6a88d2c4f74aeeaa40dbfb6b3537e79cb51fa/json HTTP/1.1" 200 None Executing query SELECT dictGetString('default.rates', 'currency', toUInt64(4990954156238030839), toDateTime('2019-10-01 00:00:00')) on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:2457 http://localhost:None "GET /v1.46/containers/8765fd477c482971969c7dca95a6a88d2c4f74aeeaa40dbfb6b3537e79cb51fa/json HTTP/1.1" 200 None ClickHouse node started Executing query CREATE DATABASE test ENGINE Memory on node Executing query CREATE TABLE test.test_table(a String) ENGINE Memory on node Stderr: Container roottestrocksdbreadonly-gw3-node-1 Stopping Stderr: Container roottestrocksdbreadonly-gw3-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/.env --project-name roottestrocksdbreadonly-gw3 --file /ClickHouse/tests/integration/test_rocksdb_read_only/_instances-0-gw3/node/docker-compose.yml down --volumes] Command:[docker compose --env-file /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/.env --project-name roottestrangehasheddictionarytypes-gw2 --file /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/node1/docker-compose.yml stop --timeout 20] [gw2] PASSED test_range_hashed_dictionary_types/test.py::test_range_hashed_dict Executing query DROP DATABASE test on node Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrestartserver-gw9-node-1 bash -c ps -C clickhouse] run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout: PID TTY TIME CMD Stdout: 10 ? 00:00:00 clickhouse run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill -9 clickhouse'] Command:[docker exec -u root roottestrestartserver-gw9-node-1 bash -c pkill -9 clickhouse] run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/storage_conf.xml) && echo PGNsaWNraG91c2U+CgogICAgPHN0b3JhZ2VfY29uZmlndXJhdGlvbj4KICAgICAgICA8ZGlza3M+CiAgICAgICAgICAgIDxzMz4KICAgICAgICAgICAgICAgIDx0eXBlPnMzPC90eXBlPgogICAgICAgICAgICAgICAgPGVuZHBvaW50Pmh0dHA6Ly9taW5pbzE6OTAwMS9yb290L2RhdGEvPC9lbmRwb2ludD4KICAgICAgICAgICAgICAgIDxhY2Nlc3Nfa2V5X2lkPm1pbmlvPC9hY2Nlc3Nfa2V5X2lkPgogICAgICAgICAgICAgICAgPHNlY3JldF9hY2Nlc3Nfa2V5Pm1pbmlvMTIzPC9zZWNyZXRfYWNjZXNzX2tleT4KICAgICAgICAgICAgICAgIDxza2lwX2FjY2Vzc19jaGVjaz50cnVlPC9za2lwX2FjY2Vzc19jaGVjaz4KICAgICAgICAgICAgPC9zMz4KICAgICAgICA8L2Rpc2tzPgogICAgICAgIDxwb2xpY2llcz4KICAgICAgICAgICAgPHMzPgogICAgICAgICAgICAgICAgPHZvbHVtZXM+CiAgICAgICAgICAgICAgICAgICAgPG1haW4+CiAgICAgICAgICAgICAgICAgICAgICAgIDxkaXNrPnMzPC9kaXNrPgogICAgICAgICAgICAgICAgICAgIDwvbWFpbj4KICAgICAgICAgICAgICAgIDwvdm9sdW1lcz4KICAgICAgICAgICAgPC9zMz4KICAgICAgICA8L3BvbGljaWVzPgogICAgPC9zdG9yYWdlX2NvbmZpZ3VyYXRpb24+CgogICAgPG1lcmdlX3RyZWU+CiAgICAgICAgPGFsbG93X3JlbW90ZV9mc196ZXJvX2NvcHlfcmVwbGljYXRpb24+MTwvYWxsb3dfcmVtb3RlX2ZzX3plcm9fY29weV9yZXBsaWNhdGlvbj4KICAgIDwvbWVyZ2VfdHJlZT4KCjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/config.d/storage_conf.xml'] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/config.d/storage_conf.xml) && echo PGNsaWNraG91c2U+CgogICAgPHN0b3JhZ2VfY29uZmlndXJhdGlvbj4KICAgICAgICA8ZGlza3M+CiAgICAgICAgICAgIDxzMz4KICAgICAgICAgICAgICAgIDx0eXBlPnMzPC90eXBlPgogICAgICAgICAgICAgICAgPGVuZHBvaW50Pmh0dHA6Ly9taW5pbzE6OTAwMS9yb290L2RhdGEvPC9lbmRwb2ludD4KICAgICAgICAgICAgICAgIDxhY2Nlc3Nfa2V5X2lkPm1pbmlvPC9hY2Nlc3Nfa2V5X2lkPgogICAgICAgICAgICAgICAgPHNlY3JldF9hY2Nlc3Nfa2V5Pm1pbmlvMTIzPC9zZWNyZXRfYWNjZXNzX2tleT4KICAgICAgICAgICAgICAgIDxza2lwX2FjY2Vzc19jaGVjaz50cnVlPC9za2lwX2FjY2Vzc19jaGVjaz4KICAgICAgICAgICAgPC9zMz4KICAgICAgICA8L2Rpc2tzPgogICAgICAgIDxwb2xpY2llcz4KICAgICAgICAgICAgPHMzPgogICAgICAgICAgICAgICAgPHZvbHVtZXM+CiAgICAgICAgICAgICAgICAgICAgPG1haW4+CiAgICAgICAgICAgICAgICAgICAgICAgIDxkaXNrPnMzPC9kaXNrPgogICAgICAgICAgICAgICAgICAgIDwvbWFpbj4KICAgICAgICAgICAgICAgIDwvdm9sdW1lcz4KICAgICAgICAgICAgPC9zMz4KICAgICAgICA8L3BvbGljaWVzPgogICAgPC9zdG9yYWdlX2NvbmZpZ3VyYXRpb24+CgogICAgPG1lcmdlX3RyZWU+CiAgICAgICAgPGFsbG93X3JlbW90ZV9mc196ZXJvX2NvcHlfcmVwbGljYXRpb24+MTwvYWxsb3dfcmVtb3RlX2ZzX3plcm9fY29weV9yZXBsaWNhdGlvbj4KICAgIDwvbWVyZ2VfdHJlZT4KCjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/config.d/storage_conf.xml] run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrestartserver-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestreplicatedzerocopyprojectionmutation-gw8-node1-1/exec HTTP/1.1" 201 74 Stdout:10 http://localhost:None "POST /v1.46/exec/d5c0e7aa6e9451754edbcdc91f089a780cd0199903ac6c1dfc5adedd353e5940/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/d5c0e7aa6e9451754edbcdc91f089a780cd0199903ac6c1dfc5adedd353e5940/json HTTP/1.1" 200 586 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Stderr: Container roottestrocksdbreadonly-gw3-node-1 Stopping Stderr: Container roottestrocksdbreadonly-gw3-node-1 Stopped Stderr: Container roottestrocksdbreadonly-gw3-node-1 Removing Stderr: Container roottestrocksdbreadonly-gw3-node-1 Removed Stderr: Network roottestrocksdbreadonly-gw3_default Removing Stderr: Network roottestrocksdbreadonly-gw3_default Removed Cleanup called Docker networks for project roottestrocksdbreadonly-gw3 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrocksdbreadonly-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrocksdbreadonly-gw3 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrocksdbreadonly-gw3-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestrocksdbreadonly-gw3 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:7 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 7 test_relative_filepath/test.py::test_filepath Running tests in /ClickHouse/tests/integration/test_relative_filepath/test.py Cluster start called. is_up=False Docker networks for project roottestrelativefilepath-gw3 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrelativefilepath-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrelativefilepath-gw3 are DRIVER VOLUME NAME Cleanup called run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Docker networks for project roottestrelativefilepath-gw3 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrelativefilepath-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Stdout:2457 Docker volumes for project roottestrelativefilepath-gw3 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrelativefilepath-gw3-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestrelativefilepath-gw3 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:7 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 7 Setup directory for instance: node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_relative_filepath/configs/config.xml'] to /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/node/database Setup logs dir /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/node/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/.env --project-name roottestrelativefilepath-gw3 --file /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/node/docker-compose.yml pull] Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrestartserver-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrestartserver-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrestartserver-gw9-node-1/exec HTTP/1.1" 201 74 Stdout:767 Clickhouse process running. run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "POST /v1.46/exec/01408a5e1797a799680bd96c8d15167fc295ff5848f2bcc124530ea69222b9a6/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/01408a5e1797a799680bd96c8d15167fc295ff5848f2bcc124530ea69222b9a6/json HTTP/1.1" 200 586 Stdout:767 Executing query select 20 on node1 run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 10 ? 00:00:00 clickhouse run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c pkill clickhouse] run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-node-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-resolver-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-node-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo3-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-minio1-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo2-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo1-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-minio1-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo1-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo3-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo2-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-resolver-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy1-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy2-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy1-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy2-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/new_node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/new_node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/switching_node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/switching_node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/.env --project-name roottestremoteblobsnamingbackwardcompatibility-gw1 --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/new_node/docker-compose.yml --file /ClickHouse/tests/integration/test_remote_blobs_naming/_instances-backward_compatibility-0-gw1/switching_node/docker-compose.yml down --volumes] Stdout:10 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:2457 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrestartserver-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:732 Clickhouse process running. run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrestartserver-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-resolver-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-node-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1 Removing Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-resolver-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-resolver-1 Removing Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-node-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-node-1 Removing Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 Removing Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-new_node-1 Removed Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-resolver-1 Removed Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-switching_node-1 Removed Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-node-1 Removed Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo2-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-minio1-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo1-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo3-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo3-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo3-1 Removing Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo2-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo2-1 Removing Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo1-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo1-1 Removing Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-minio1-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-minio1-1 Removing Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo3-1 Removed Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo2-1 Removed Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-zoo1-1 Removed Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-minio1-1 Removed Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy1-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy2-1 Stopping Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy1-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy1-1 Removing Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy2-1 Stopped Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy2-1 Removing Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy2-1 Removed Stderr: Container roottestremoteblobsnamingbackwardcompatibility-gw1-proxy1-1 Removed Stderr: Volume roottestremoteblobsnamingbackwardcompatibility-gw1_data1-1 Removing Stderr: Network roottestremoteblobsnamingbackwardcompatibility-gw1_default Removing Stderr: Volume roottestremoteblobsnamingbackwardcompatibility-gw1_data1-1 Removed Stderr: Network roottestremoteblobsnamingbackwardcompatibility-gw1_default Removed Cleanup called Docker networks for project roottestremoteblobsnamingbackwardcompatibility-gw1 are NETWORK ID NAME DRIVER SCOPE Stdout:732 Executing query select 20 on node Docker containers for project roottestremoteblobsnamingbackwardcompatibility-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestremoteblobsnamingbackwardcompatibility-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestremoteblobsnamingbackwardcompatibility-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestremoteblobsnamingbackwardcompatibility-gw1 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Executing query SHOW DATABASES LIKE 'test' on node Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:5 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 5 Running tests in /ClickHouse/tests/integration/test_recovery_time_metric/test.py test_recovery_time_metric/test.py::test_recovery_time_metric Cluster start called. is_up=False [gw9] PASSED test_restart_server/test.py::test_drop_memory_database Executing query CREATE TABLE flush_test (a String, b UInt64) ENGINE = MergeTree ORDER BY a; SET async_insert = 1; SET wait_for_async_insert = 0; SET async_insert_busy_timeout_ms = 1000000; INSERT INTO flush_test VALUES ('world', 23456); on node test_restart_server/test.py::test_flushes_async_insert_queue Docker networks for project roottestrecoverytimemetric-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrecoverytimemetric-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrecoverytimemetric-gw1 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestrecoverytimemetric-gw1 are NETWORK ID NAME DRIVER SCOPE run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrestartserver-gw9-node-1 bash -c ps -C clickhouse] Docker containers for project roottestrecoverytimemetric-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrecoverytimemetric-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrecoverytimemetric-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestrecoverytimemetric-gw1 Trying to prune unused networks... run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout: PID TTY TIME CMD Stdout: 732 ? 00:00:00 clickhouse run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrestartserver-gw9-node-1 bash -c pkill clickhouse] Trying to prune unused images... Command:[docker image prune -f] run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrestartserver-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:Total reclaimed space: 0B Stdout:10 Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:5 Command:[docker volume prune -f] Stdout:732 Stdout:Total reclaimed space: 0B Volumes pruned: 5 Setup directory for instance: node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_recovery_time_metric/configs/config.xml'] to /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/node/database Setup logs dir /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/node/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/.env --project-name roottestrecoverytimemetric-gw1 --file /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/node/docker-compose.yml pull] Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:2457 Stderr: Container roottestrangehasheddictionarytypes-gw2-node1-1 Stopping Stderr: Container roottestrangehasheddictionarytypes-gw2-node1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/.env --project-name roottestrangehasheddictionarytypes-gw2 --file /ClickHouse/tests/integration/test_range_hashed_dictionary_types/_instances-0-gw2/node1/docker-compose.yml down --volumes] Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Stderr: Container roottestrangehasheddictionarytypes-gw2-node1-1 Stopping Stderr: Container roottestrangehasheddictionarytypes-gw2-node1-1 Stopped Stderr: Container roottestrangehasheddictionarytypes-gw2-node1-1 Removing Stderr: Container roottestrangehasheddictionarytypes-gw2-node1-1 Removed Stderr: Network roottestrangehasheddictionarytypes-gw2_default Removing Stderr: Network roottestrangehasheddictionarytypes-gw2_default Removed Cleanup called Docker networks for project roottestrangehasheddictionarytypes-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrangehasheddictionarytypes-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrangehasheddictionarytypes-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrangehasheddictionarytypes-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestrangehasheddictionarytypes-gw2 Trying to prune unused networks... run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrestartserver-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:10 Stdout:5 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 5 test_render_log_file_name_templates/test.py::test_check_file_names clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} Cluster name: project_name:roottestrenderlogfilenametemplates-gw2. Added instance name:file-names-from-config tag:8b2301119731 base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/.env', '--project-name', 'roottestrenderlogfilenametemplates-gw2', '--file', '/ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-config/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server-%Y-%m.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server-%Y-%m.err.log Cluster name: project_name:roottestrenderlogfilenametemplates-gw2. Added instance name:file-names-from-params tag:8b2301119731 base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/.env', '--project-name', 'roottestrenderlogfilenametemplates-gw2', '--file', '/ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-config/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-params/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ Running tests in /ClickHouse/tests/integration/test_render_log_file_name_templates/test.py Cluster start called. is_up=False Stdout:732 Docker networks for project roottestrenderlogfilenametemplates-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrenderlogfilenametemplates-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrenderlogfilenametemplates-gw2 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestrenderlogfilenametemplates-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrenderlogfilenametemplates-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrenderlogfilenametemplates-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrenderlogfilenametemplates-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestrenderlogfilenametemplates-gw2 Trying to prune unused networks... run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Trying to prune unused images... Command:[docker image prune -f] Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:2457 Stdout:5 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 5 Setup directory for instance: file-names-from-config Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_render_log_file_name_templates/configs/config-file-template.xml'] to /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-config/configs/config.d Setup database dir /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-config/database Setup logs dir /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-config/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--"] Setup directory for instance: file-names-from-params Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-params/configs/config.d Setup database dir /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-params/database Setup logs dir /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-params/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server-%Y-%m.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server-%Y-%m.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/.env --project-name roottestrenderlogfilenametemplates-gw2 --file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-config/docker-compose.yml --file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-params/docker-compose.yml pull] Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrestartserver-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/storage_conf.xml) && echo PGNsaWNraG91c2U+CgogICAgPHN0b3JhZ2VfY29uZmlndXJhdGlvbj4KICAgICAgICA8ZGlza3M+CiAgICAgICAgICAgIDxzMz4KICAgICAgICAgICAgICAgIDx0eXBlPnMzPC90eXBlPgogICAgICAgICAgICAgICAgPGVuZHBvaW50Pmh0dHA6Ly9taW5pbzE6OTAwMS9yb290L2RhdGEvPC9lbmRwb2ludD4KICAgICAgICAgICAgICAgIDxhY2Nlc3Nfa2V5X2lkPm1pbmlvPC9hY2Nlc3Nfa2V5X2lkPgogICAgICAgICAgICAgICAgPHNlY3JldF9hY2Nlc3Nfa2V5Pm1pbmlvMTIzPC9zZWNyZXRfYWNjZXNzX2tleT4KICAgICAgICAgICAgICAgIDxza2lwX2FjY2Vzc19jaGVjaz50cnVlPC9za2lwX2FjY2Vzc19jaGVjaz4KICAgICAgICAgICAgPC9zMz4KICAgICAgICA8L2Rpc2tzPgogICAgICAgIDxwb2xpY2llcz4KICAgICAgICAgICAgPHMzPgogICAgICAgICAgICAgICAgPHZvbHVtZXM+CiAgICAgICAgICAgICAgICAgICAgPG1haW4+CiAgICAgICAgICAgICAgICAgICAgICAgIDxkaXNrPnMzPC9kaXNrPgogICAgICAgICAgICAgICAgICAgIDwvbWFpbj4KICAgICAgICAgICAgICAgIDwvdm9sdW1lcz4KICAgICAgICAgICAgPC9zMz4KICAgICAgICA8L3BvbGljaWVzPgogICAgPC9zdG9yYWdlX2NvbmZpZ3VyYXRpb24+CgogICAgPG1lcmdlX3RyZWU+CiAgICAgICAgPGFsbG93X3JlbW90ZV9mc196ZXJvX2NvcHlfcmVwbGljYXRpb24+MTwvYWxsb3dfcmVtb3RlX2ZzX3plcm9fY29weV9yZXBsaWNhdGlvbj4KICAgIDwvbWVyZ2VfdHJlZT4KCjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/config.d/storage_conf.xml'] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/config.d/storage_conf.xml) && echo PGNsaWNraG91c2U+CgogICAgPHN0b3JhZ2VfY29uZmlndXJhdGlvbj4KICAgICAgICA8ZGlza3M+CiAgICAgICAgICAgIDxzMz4KICAgICAgICAgICAgICAgIDx0eXBlPnMzPC90eXBlPgogICAgICAgICAgICAgICAgPGVuZHBvaW50Pmh0dHA6Ly9taW5pbzE6OTAwMS9yb290L2RhdGEvPC9lbmRwb2ludD4KICAgICAgICAgICAgICAgIDxhY2Nlc3Nfa2V5X2lkPm1pbmlvPC9hY2Nlc3Nfa2V5X2lkPgogICAgICAgICAgICAgICAgPHNlY3JldF9hY2Nlc3Nfa2V5Pm1pbmlvMTIzPC9zZWNyZXRfYWNjZXNzX2tleT4KICAgICAgICAgICAgICAgIDxza2lwX2FjY2Vzc19jaGVjaz50cnVlPC9za2lwX2FjY2Vzc19jaGVjaz4KICAgICAgICAgICAgPC9zMz4KICAgICAgICA8L2Rpc2tzPgogICAgICAgIDxwb2xpY2llcz4KICAgICAgICAgICAgPHMzPgogICAgICAgICAgICAgICAgPHZvbHVtZXM+CiAgICAgICAgICAgICAgICAgICAgPG1haW4+CiAgICAgICAgICAgICAgICAgICAgICAgIDxkaXNrPnMzPC9kaXNrPgogICAgICAgICAgICAgICAgICAgIDwvbWFpbj4KICAgICAgICAgICAgICAgIDwvdm9sdW1lcz4KICAgICAgICAgICAgPC9zMz4KICAgICAgICA8L3BvbGljaWVzPgogICAgPC9zdG9yYWdlX2NvbmZpZ3VyYXRpb24+CgogICAgPG1lcmdlX3RyZWU+CiAgICAgICAgPGFsbG93X3JlbW90ZV9mc196ZXJvX2NvcHlfcmVwbGljYXRpb24+MTwvYWxsb3dfcmVtb3RlX2ZzX3plcm9fY29weV9yZXBsaWNhdGlvbj4KICAgIDwvbWVyZ2VfdHJlZT4KCjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/config.d/storage_conf.xml] Stdout:732 run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestreplicatedzerocopyprojectionmutation-gw8-node2-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/0bfb1f687e98f8ba580867ba473413a0a718fc52f6331ee396eab538975d4685/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/0bfb1f687e98f8ba580867ba473413a0a718fc52f6331ee396eab538975d4685/json HTTP/1.1" 200 586 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:2457 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrestartserver-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrestartserver-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrestartserver-gw9-node-1/exec HTTP/1.1" 201 74 run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "POST /v1.46/exec/0205cd38525dd8a9ae080f407b57b3b4b8475b15678a4092cdaae0bf617076f4/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/0205cd38525dd8a9ae080f407b57b3b4b8475b15678a4092cdaae0bf617076f4/json HTTP/1.1" 200 586 Stdout:769 Clickhouse process running. run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:769 Executing query select 20 on node2 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/exec/1f3b5214f823f0a0ea396a8bff8f3a98ae1434661385fa9f55a6441c68c142f3/json HTTP/1.1" 200 584 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://172.16.5.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=data%2F HTTP/1.1" 200 0 list_objects (0): [] Executing query DROP TABLE IF EXISTS test_all_projection_files_are_dropped SYNC on node1 No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrefreshablemv-gw5-node1-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/752ba1e35875bbf39a899dc23c9528c9bfe59482d41075096fee4fff2b1a49ab/start HTTP/1.1" 200 0 Executing query CREATE TABLE test_all_projection_files_are_dropped(a UInt32, b UInt32) ENGINE MergeTree() ORDER BY a SETTINGS storage_policy='s3', old_parts_lifetime=0 on node1 http://localhost:None "GET /v1.46/exec/752ba1e35875bbf39a899dc23c9528c9bfe59482d41075096fee4fff2b1a49ab/json HTTP/1.1" 200 586 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Executing query ALTER TABLE test_all_projection_files_are_dropped ADD projection b_order (SELECT a, b ORDER BY b) on node1 Executing query ALTER TABLE test_all_projection_files_are_dropped MATERIALIZE projection b_order on node1 http://172.16.5.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=data%2F HTTP/1.1" 200 0 list_objects (2): ['data/ewc/hwalvhqhfdcmvwarxtkwnjrihgvtg', 'data/kys/qqtmiixyjyhdtlzrrslcgrzbbumbe'] Executing query INSERT INTO test_all_projection_files_are_dropped VALUES (1, 105), (5, 101), (3, 103), (4, 102), (2, 104) on node1 Executing query ALTER TABLE test_all_projection_files_are_dropped DROP PARTITION ID 'all' on node1 http://172.16.5.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=data%2F HTTP/1.1" 200 0 list_objects (2): ['data/ewc/hwalvhqhfdcmvwarxtkwnjrihgvtg', 'data/kys/qqtmiixyjyhdtlzrrslcgrzbbumbe'] Executing query DROP TABLE IF EXISTS test_all_projection_files_are_dropped SYNC on node1 [gw8] PASSED test_replicated_zero_copy_projection_mutation/test.py::test_all_projection_files_are_dropped_when_part_is_dropped test_replicated_zero_copy_projection_mutation/test.py::test_hardlinks_preserved_when_projection_dropped http://172.16.5.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=data%2F HTTP/1.1" 200 0 list_objects (0): [] Executing query DROP TABLE IF EXISTS test_hardlinks_preserved_when_projection_dropped SYNC on node1 Executing query DROP TABLE IF EXISTS test_hardlinks_preserved_when_projection_dropped SYNC on node2 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Executing query CREATE TABLE test_hardlinks_preserved_when_projection_dropped ( a UInt32, b UInt32, c UInt32, PROJECTION projection_order_by_b ( SELECT a, b ORDER BY b ) ) ENGINE ReplicatedMergeTree('/clickhouse/tables/test_projection', '{instance}') ORDER BY a SETTINGS cleanup_delay_period=1, max_cleanup_delay_period=3 , storage_policy='s3', old_parts_lifetime=0 on node1 Executing query CREATE TABLE test_hardlinks_preserved_when_projection_dropped ( a UInt32, b UInt32, c UInt32, PROJECTION projection_order_by_b ( SELECT a, b ORDER BY b ) ) ENGINE ReplicatedMergeTree('/clickhouse/tables/test_projection', '{instance}') ORDER BY a SETTINGS cleanup_delay_period=1, max_cleanup_delay_period=3 , storage_policy='s3', old_parts_lifetime=10000 on node2 run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrestartserver-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://172.16.5.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=data%2F HTTP/1.1" 200 0 list_objects (2): ['data/htw/tvdbubwgquyarlkmaycdtjvkufopo', 'data/nfd/ryldrcydnezxszamqcuffozixsxqd'] Executing query SYSTEM FLUSH LOGS on node1 Stdout:1507 Clickhouse process running. run container_id:roottestrestartserver-gw9-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrestartserver-gw9-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:1507 Executing query select 20 on node Executing query SELECT * FROM flush_test on node Executing query SELECT uuid FROM system.tables WHERE name = 'test_hardlinks_preserved_when_projection_dropped' on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Command:[docker compose --env-file /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/.env --project-name roottestrestartserver-gw9 --file /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/node/docker-compose.yml stop --timeout 20] [gw9] PASSED test_restart_server/test.py::test_flushes_async_insert_queue Executing query INSERT INTO test_hardlinks_preserved_when_projection_dropped VALUES (1, 105, 1), (5, 101, 1), (3, 103, 1), (4, 102, 1), (2, 104, 1) on node1 Stdout:3346 Clickhouse process running. run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:3346 Executing query select 20 on node1 Executing query SYSTEM STOP MERGES on node2 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c ps -C clickhouse] Executing query ALTER TABLE test_hardlinks_preserved_when_projection_dropped UPDATE c = 2 where c = 1 on node1 Stdout: PID TTY TIME CMD Stdout: 3346 ? 00:00:01 clickhouse run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c pkill clickhouse] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:3346 Executing query SELECT COUNT() FROM system.replication_queue on node1 Stderr: Container roottestrestartserver-gw9-node-1 Stopping Stderr: Container roottestrestartserver-gw9-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/.env --project-name roottestrestartserver-gw9 --file /ClickHouse/tests/integration/test_restart_server/_instances-0-gw9/node/docker-compose.yml down --volumes] Executing query SYSTEM START MERGES on node2 Executing query SELECT removal_state FROM system.parts WHERE name = 'all_0_0_0' AND table = 'test_hardlinks_preserved_when_projection_dropped' AND not active on node1 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Stderr: Container roottestrestartserver-gw9-node-1 Stopping Stderr: Container roottestrestartserver-gw9-node-1 Stopped Stderr: Container roottestrestartserver-gw9-node-1 Removing Stderr: Container roottestrestartserver-gw9-node-1 Removed Stderr: Network roottestrestartserver-gw9_default Removing Stderr: Network roottestrestartserver-gw9_default Removed Cleanup called Docker networks for project roottestrestartserver-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrestartserver-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrestartserver-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrestartserver-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestrestartserver-gw9 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:5 Command:[docker volume prune -f] Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Stdout:Total reclaimed space: 0B Volumes pruned: 5 test_replica_can_become_leader/test.py::test_can_become_leader Running tests in /ClickHouse/tests/integration/test_replica_can_become_leader/test.py Cluster start called. is_up=False Docker networks for project roottestreplicacanbecomeleader-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicacanbecomeleader-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreplicacanbecomeleader-gw9 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestreplicacanbecomeleader-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicacanbecomeleader-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Docker volumes for project roottestreplicacanbecomeleader-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicacanbecomeleader-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreplicacanbecomeleader-gw9 Trying to prune unused networks... Stdout:3346 Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:5 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 5 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_replica_can_become_leader/configs/notleader.xml'] to /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node1/database Setup logs dir /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_replica_can_become_leader/configs/notleaderignorecase.xml'] to /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node2/database Setup logs dir /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node3 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node3/configs/config.d Setup database dir /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node3/database Setup logs dir /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node3/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/.env --project-name roottestreplicacanbecomeleader-gw9 --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node3/docker-compose.yml pull] Executing query SELECT removal_state FROM system.parts WHERE name = 'all_0_0_0' AND table = 'test_hardlinks_preserved_when_projection_dropped' AND not active on node1 Executing query SELECT value FROM system.zookeeper WHERE path like '/clickhouse/zero_copy/zero_copy_s3/16f7d6e8-ddf0-4f72-ab6e-747f08255458' AND name = 'all_0_0_0' on node1 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Executing query SELECT path FROM system.parts WHERE name = 'all_0_0_0' AND table = 'test_hardlinks_preserved_when_projection_dropped' on node2 run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', 'INDEX_FILE=/var/lib/clickhouse/disks/s3/store/4a8/4a88ba76-6290-418c-adcc-8a47d23b9357/all_0_0_0//primary.cidx\n cp $INDEX_FILE $INDEX_FILE.backup\n echo "unexpected data in metadata file" | cat > $INDEX_FILE\n '] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c INDEX_FILE=/var/lib/clickhouse/disks/s3/store/4a8/4a88ba76-6290-418c-adcc-8a47d23b9357/all_0_0_0//primary.cidx cp $INDEX_FILE $INDEX_FILE.backup echo "unexpected data in metadata file" | cat > $INDEX_FILE ] run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 769 ? 00:00:00 clickhouse run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c pkill clickhouse] run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:769 Stderr: node1 Skipped - Image is already being pulled by node3 Stderr: zoo1 Skipped - Image is already being pulled by node3 Stderr: zoo2 Skipped - Image is already being pulled by node3 Stderr: zoo3 Skipped - Image is already being pulled by node3 Stderr: node2 Skipped - Image is already being pulled by node3 Stderr: node3 Pulling Stderr: node3 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper1/log', '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper1/config', '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper1/coordination', '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper2/log', '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper2/config', '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper2/coordination', '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper3/log', '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper3/config', '/ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/keeper3/coordination'] Command:[docker compose --project-name roottestreplicacanbecomeleader-gw9 --env-file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr: node Pulling Stderr: node Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/.env --project-name roottestrecoverytimemetric-gw1 --file /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/.env --project-name roottestrecoverytimemetric-gw1 --file /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/node/docker-compose.yml up -d --no-recreate] Stderr: node Pulling Stderr: node Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/.env --project-name roottestrelativefilepath-gw3 --file /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/.env --project-name roottestrelativefilepath-gw3 --file /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/node/docker-compose.yml up -d --no-recreate] Stderr: file-names-from-params Skipped - Image is already being pulled by file-names-from-config Stderr: file-names-from-config Pulling Stderr: file-names-from-config Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/.env --project-name roottestrenderlogfilenametemplates-gw2 --file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-config/docker-compose.yml --file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-params/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/.env --project-name roottestrenderlogfilenametemplates-gw2 --file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-config/docker-compose.yml --file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-params/docker-compose.yml up -d --no-recreate] Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stderr: node1 Skipped - Image is already being pulled by node2 Stderr: node2 Pulling Stderr: node2 Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/.env --project-name roottestprofilesettingsandconstraintsorder-gw0 --file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/.env --project-name roottestprofilesettingsandconstraintsorder-gw0 --file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node2/docker-compose.yml up -d --no-recreate] Stdout:3346 Stderr:time="2025-04-02T03:59:20Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestreplicacanbecomeleader-gw9_default Creating Stderr: Network roottestreplicacanbecomeleader-gw9_default Created Stderr: Container roottestreplicacanbecomeleader-gw9-zoo2-1 Creating Stderr: Container roottestreplicacanbecomeleader-gw9-zoo3-1 Creating Stderr: Container roottestreplicacanbecomeleader-gw9-zoo1-1 Creating Stderr: Container roottestreplicacanbecomeleader-gw9-zoo3-1 Created Stderr: Container roottestreplicacanbecomeleader-gw9-zoo2-1 Created Stderr: Container roottestreplicacanbecomeleader-gw9-zoo1-1 Created Stderr: Container roottestreplicacanbecomeleader-gw9-zoo1-1 Starting Stderr: Container roottestreplicacanbecomeleader-gw9-zoo2-1 Starting Stderr: Container roottestreplicacanbecomeleader-gw9-zoo3-1 Starting Stderr: Container roottestreplicacanbecomeleader-gw9-zoo1-1 Started Stderr: Container roottestreplicacanbecomeleader-gw9-zoo2-1 Started Stderr: Container roottestreplicacanbecomeleader-gw9-zoo3-1 Started Stderr:time="2025-04-02T03:59:20Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:59:20Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestreplicacanbecomeleader-gw9-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.1.2, port:2181, use_ssl:False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Stderr: Network roottestrecoverytimemetric-gw1_default Creating Stderr: Network roottestrecoverytimemetric-gw1_default Created Stderr: Container roottestrecoverytimemetric-gw1-node-1 Creating Stderr: Container roottestrecoverytimemetric-gw1-node-1 Created Stderr: Container roottestrecoverytimemetric-gw1-node-1 Starting Stderr: Container roottestrecoverytimemetric-gw1-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestrecoverytimemetric-gw1-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestrecoverytimemetric-gw1-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.3.2... http://localhost:None "GET /v1.46/containers/roottestrecoverytimemetric-gw1-node-1/json HTTP/1.1" 200 None Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None Stdout:769 http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Stderr: Network roottestrelativefilepath-gw3_default Creating Stderr: Network roottestrelativefilepath-gw3_default Created Stderr: Container roottestrelativefilepath-gw3-node-1 Creating Stderr: Container roottestrelativefilepath-gw3-node-1 Created Stderr: Container roottestrelativefilepath-gw3-node-1 Starting Stderr: Container roottestrelativefilepath-gw3-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestrelativefilepath-gw3-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestrelativefilepath-gw3-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.4.2... http://localhost:None "GET /v1.46/containers/roottestrelativefilepath-gw3-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cc2e5a239634391395a3335702ceb37882ff6c512da392754e1146e72fe5ff3c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cc2e5a239634391395a3335702ceb37882ff6c512da392754e1146e72fe5ff3c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None Stderr: Network roottestrenderlogfilenametemplates-gw2_default Creating Stderr: Network roottestrenderlogfilenametemplates-gw2_default Created Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 Creating Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 Creating Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 Created Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 Created Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 Starting Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 Starting Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 Started Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 Started ClickHouse instance created get_instance_ip instance_name=file-names-from-config http://localhost:None "GET /v1.46/containers/roottestrenderlogfilenametemplates-gw2-file-names-from-config-1/json HTTP/1.1" 200 None get_instance_ip instance_name=file-names-from-config run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] http://localhost:None "GET /v1.46/containers/roottestrenderlogfilenametemplates-gw2-file-names-from-config-1/json HTTP/1.1" 200 None Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Waiting for ClickHouse start in file-names-from-config, ip: 172.16.8.3... http://localhost:None "GET /v1.46/containers/roottestrenderlogfilenametemplates-gw2-file-names-from-config-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/8b50d4dee433b827f67fac3ad6f56bcb593d2c9f448e533b26e2c28fada64c46/json HTTP/1.1" 200 None Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Stdout:3346 http://localhost:None "GET /v1.46/containers/cc2e5a239634391395a3335702ceb37882ff6c512da392754e1146e72fe5ff3c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/8b50d4dee433b827f67fac3ad6f56bcb593d2c9f448e533b26e2c28fada64c46/json HTTP/1.1" 200 None Stderr: Network roottestprofilesettingsandconstraintsorder-gw0_default Creating Stderr: Network roottestprofilesettingsandconstraintsorder-gw0_default Created Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node2-1 Creating Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node1-1 Creating Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node2-1 Created Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node1-1 Created Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node2-1 Starting Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node1-1 Starting Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node1-1 Started Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node2-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestprofilesettingsandconstraintsorder-gw0-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestprofilesettingsandconstraintsorder-gw0-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.10.2... http://localhost:None "GET /v1.46/containers/roottestprofilesettingsandconstraintsorder-gw0-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/7d727727da0e803a4b75b66d4b13538d0377683b02619eb22514a832f0d5c95d/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cc2e5a239634391395a3335702ceb37882ff6c512da392754e1146e72fe5ff3c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/8b50d4dee433b827f67fac3ad6f56bcb593d2c9f448e533b26e2c28fada64c46/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/7d727727da0e803a4b75b66d4b13538d0377683b02619eb22514a832f0d5c95d/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cc2e5a239634391395a3335702ceb37882ff6c512da392754e1146e72fe5ff3c/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/8b50d4dee433b827f67fac3ad6f56bcb593d2c9f448e533b26e2c28fada64c46/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/7d727727da0e803a4b75b66d4b13538d0377683b02619eb22514a832f0d5c95d/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/cc2e5a239634391395a3335702ceb37882ff6c512da392754e1146e72fe5ff3c/json HTTP/1.1" 200 None ClickHouse node started run container_id:roottestrelativefilepath-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p user_files'] Command:[docker exec -u root --privileged roottestrelativefilepath-gw3-node-1 bash -c mkdir -p user_files] http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/8b50d4dee433b827f67fac3ad6f56bcb593d2c9f448e533b26e2c28fada64c46/json HTTP/1.1" 200 None ClickHouse file-names-from-config started get_instance_ip instance_name=file-names-from-params http://localhost:None "GET /v1.46/containers/roottestrenderlogfilenametemplates-gw2-file-names-from-params-1/json HTTP/1.1" 200 None get_instance_ip instance_name=file-names-from-params run container_id:roottestrelativefilepath-gw3-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'echo "Test\t111.222\nData\t333.444" > user_files/relative_user_file_test'] Command:[docker exec -u root --privileged roottestrelativefilepath-gw3-node-1 bash -c echo "Test 111.222 Data 333.444" > user_files/relative_user_file_test] http://localhost:None "GET /v1.46/containers/roottestrenderlogfilenametemplates-gw2-file-names-from-params-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in file-names-from-params, ip: 172.16.8.2... http://localhost:None "GET /v1.46/containers/roottestrenderlogfilenametemplates-gw2-file-names-from-params-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/06b3b7894cbac626cb4b45de92a4b53f216f8ad49d879e3b3aaa963ddd36f389/json HTTP/1.1" 200 None ClickHouse file-names-from-params started log_file /var/log/clickhouse-server/clickhouse-server-2025-04.log err_log_file /var/log/clickhouse-server/clickhouse-server-2025-04.err.log run container_id:roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 detach:False nothrow:True cmd: ['bash', '-c', 'ls -lh /var/log/clickhouse-server/'] Command:[docker exec roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 bash -c ls -lh /var/log/clickhouse-server/] http://localhost:None "GET /v1.46/containers/7d727727da0e803a4b75b66d4b13538d0377683b02619eb22514a832f0d5c95d/json HTTP/1.1" 200 None Executing query select count() from file('relative_user_file_test', 'TSV', 'text String, number Float64') on node Stdout:total 56K Stdout:-rw-r----- 1 root root 632 Apr 2 03:59 clickhouse-server-2025-04.err.log Stdout:-rw-r----- 1 root root 45K Apr 2 03:59 clickhouse-server-2025-04.log http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None Stdout:-rw------- 1 root root 152 Apr 2 03:59 stderr.log Stdout:-rw-r----- 1 root root 0 Apr 2 03:59 stdout.log check instance 'file-names-from-config': /var/log/clickhouse-server/ contains: total 56K -rw-r----- 1 root root 632 Apr 2 03:59 clickhouse-server-2025-04.err.log -rw-r----- 1 root root 45K Apr 2 03:59 clickhouse-server-2025-04.log -rw------- 1 root root 152 Apr 2 03:59 stderr.log -rw-r----- 1 root root 0 Apr 2 03:59 stdout.log run container_id:roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 detach:False nothrow:True cmd: ['bash', '-c', 'ls /var/log/clickhouse-server/clickhouse-server-2025-04.log'] Command:[docker exec roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 bash -c ls /var/log/clickhouse-server/clickhouse-server-2025-04.log] Stdout:/var/log/clickhouse-server/clickhouse-server-2025-04.log run container_id:roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 detach:False nothrow:True cmd: ['bash', '-c', 'ls /var/log/clickhouse-server/clickhouse-server-2025-04.err.log'] Command:[docker exec roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 bash -c ls /var/log/clickhouse-server/clickhouse-server-2025-04.err.log] http://localhost:None "GET /v1.46/containers/7d727727da0e803a4b75b66d4b13538d0377683b02619eb22514a832f0d5c95d/json HTTP/1.1" 200 None Stdout:/var/log/clickhouse-server/clickhouse-server-2025-04.err.log run container_id:roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 detach:False nothrow:True cmd: ['bash', '-c', 'ls -lh /var/log/clickhouse-server/'] Command:[docker exec roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 bash -c ls -lh /var/log/clickhouse-server/] Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query select count() from file('../user_files/relative_user_file_test', 'TSV', 'text String, number Float64') on node http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None Stdout:total 56K Stdout:-rw-r----- 1 root root 632 Apr 2 03:59 clickhouse-server-2025-04.err.log Stdout:-rw-r----- 1 root root 45K Apr 2 03:59 clickhouse-server-2025-04.log Stdout:-rw------- 1 root root 152 Apr 2 03:59 stderr.log Stdout:-rw-r----- 1 root root 0 Apr 2 03:59 stdout.log check instance 'file-names-from-params': /var/log/clickhouse-server/ contains: total 56K -rw-r----- 1 root root 632 Apr 2 03:59 clickhouse-server-2025-04.err.log -rw-r----- 1 root root 45K Apr 2 03:59 clickhouse-server-2025-04.log -rw------- 1 root root 152 Apr 2 03:59 stderr.log -rw-r----- 1 root root 0 Apr 2 03:59 stdout.log run container_id:roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 detach:False nothrow:True cmd: ['bash', '-c', 'ls /var/log/clickhouse-server/clickhouse-server-2025-04.log'] Command:[docker exec roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 bash -c ls /var/log/clickhouse-server/clickhouse-server-2025-04.log] http://localhost:None "GET /v1.46/containers/7d727727da0e803a4b75b66d4b13538d0377683b02619eb22514a832f0d5c95d/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestprofilesettingsandconstraintsorder-gw0-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestprofilesettingsandconstraintsorder-gw0-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.10.3... http://localhost:None "GET /v1.46/containers/roottestprofilesettingsandconstraintsorder-gw0-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/611e9f7824eba9eb26288005619204c5965802d1109cbc2bf6b0fc308acaabd3/json HTTP/1.1" 200 None ClickHouse node2 started Executing query SELECT name, readonly FROM system.settings WHERE name == 'log_queries' on node1 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Stdout:/var/log/clickhouse-server/clickhouse-server-2025-04.log run container_id:roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 detach:False nothrow:True cmd: ['bash', '-c', 'ls /var/log/clickhouse-server/clickhouse-server-2025-04.err.log'] Command:[docker exec roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 bash -c ls /var/log/clickhouse-server/clickhouse-server-2025-04.err.log] http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Command:[docker compose --env-file /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/.env --project-name roottestrelativefilepath-gw3 --file /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/node/docker-compose.yml stop --timeout 20] [gw3] PASSED test_relative_filepath/test.py::test_filepath Stdout:/var/log/clickhouse-server/clickhouse-server-2025-04.err.log Command:[docker compose --env-file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/.env --project-name roottestrenderlogfilenametemplates-gw2 --file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-config/docker-compose.yml --file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-params/docker-compose.yml stop --timeout 20] [gw2] PASSED test_render_log_file_name_templates/test.py::test_check_file_names run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestreplicatedzerocopyprojectionmutation-gw8-node2-1/exec HTTP/1.1" 201 74 Executing query SELECT name, readonly FROM system.settings WHERE name == 'log_queries' on node2 http://localhost:None "POST /v1.46/exec/bd7b4763c5a00660c74a01ce4ec88321d1fd650ccbf2b52a353a3296c3abc558/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/bd7b4763c5a00660c74a01ce4ec88321d1fd650ccbf2b52a353a3296c3abc558/json HTTP/1.1" 200 586 Command:[docker compose --env-file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/.env --project-name roottestprofilesettingsandconstraintsorder-gw0 --file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node2/docker-compose.yml stop --timeout 20] [gw0] PASSED test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/exec/752ba1e35875bbf39a899dc23c9528c9bfe59482d41075096fee4fff2b1a49ab/json HTTP/1.1" 200 584 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrefreshablemv-gw5-node1-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/31b807f35f1177487aaee6679e959c84f2702eaeba60faa1e2c9faa064913cb3/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/31b807f35f1177487aaee6679e959c84f2702eaeba60faa1e2c9faa064913cb3/json HTTP/1.1" 200 586 http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 http://localhost:None "GET /v1.46/containers/c21dd4f9a624764eca9ce2d7edaa72b3b169a86a32e93a81d5e4d4102c27befa/json HTTP/1.1" 200 None ClickHouse node started Executing query DROP DATABASE IF EXISTS rdb; CREATE DATABASE rdb ENGINE = Replicated('/test/test_recovery_time_metric', 'shard1', 'replica1') on node run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query DROP TABLE IF EXISTS rdb.t; CREATE TABLE rdb.t ( `x` UInt32 ) ENGINE = MergeTree ORDER BY x on node Stdout:1547 Clickhouse process running. run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:1547 Executing query select 20 on node2 run container_id:roottestrecoverytimemetric-gw1-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'rm /var/lib/clickhouse/metadata/rdb/t.sql'] Command:[docker exec roottestrecoverytimemetric-gw1-node-1 bash -c rm /var/lib/clickhouse/metadata/rdb/t.sql] Executing query SYSTEM WAIT LOADING PARTS test_hardlinks_preserved_when_projection_dropped on node2 run container_id:roottestrecoverytimemetric-gw1-node-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrecoverytimemetric-gw1-node-1 bash -c ps -C clickhouse] Executing query SYSTEM FLUSH LOGS on node2 Stdout: PID TTY TIME CMD Stdout: 10 ? 00:00:00 clickhouse run container_id:roottestrecoverytimemetric-gw1-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrecoverytimemetric-gw1-node-1 bash -c pkill clickhouse] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestrecoverytimemetric-gw1-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrecoverytimemetric-gw1-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:4157 Clickhouse process running. run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:10 Stdout:4157 Executing query select 20 on node1 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Executing query SELECT name, reason, path FROM system.detached_parts WHERE table = 'test_hardlinks_preserved_when_projection_dropped' on node2 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c ps -C clickhouse] run container_id:roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 detach:False nothrow:False cmd: ['bash', '-c', 'INDEX_FILE=/var/lib/clickhouse/disks/s3/store/4a8/4a88ba76-6290-418c-adcc-8a47d23b9357/detached/broken-on-start_all_0_0_0/primary.cidx\n mv $INDEX_FILE.backup $INDEX_FILE\n '] Command:[docker exec roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 bash -c INDEX_FILE=/var/lib/clickhouse/disks/s3/store/4a8/4a88ba76-6290-418c-adcc-8a47d23b9357/detached/broken-on-start_all_0_0_0/primary.cidx mv $INDEX_FILE.backup $INDEX_FILE ] Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Stdout: PID TTY TIME CMD Stdout: 4157 ? 00:00:00 clickhouse run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c pkill clickhouse] Executing query ALTER TABLE test_hardlinks_preserved_when_projection_dropped DROP DETACHED PART 'broken-on-start_all_0_0_0' on node2 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:4157 Executing query CHECK TABLE test_hardlinks_preserved_when_projection_dropped on node1 Executing query CHECK TABLE test_hardlinks_preserved_when_projection_dropped on node2 Executing query ALTER TABLE test_hardlinks_preserved_when_projection_dropped DROP PART 'all_0_0_0_1' on node2 Stderr: Container roottestrelativefilepath-gw3-node-1 Stopping Stderr: Container roottestrelativefilepath-gw3-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/.env --project-name roottestrelativefilepath-gw3 --file /ClickHouse/tests/integration/test_relative_filepath/_instances-0-gw3/node/docker-compose.yml down --volumes] Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 http://172.16.5.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=data%2F HTTP/1.1" 200 0 list_objects (18): ['data/dom/yusxjhjmhxyceahdmqqocrbmgytgb', 'data/dsm/abzesqkqilfkwoiuqfqykqczzwmkb', 'data/eox/rdlzgcwqyvhwlhwspdwqjdyxiigkk', 'data/erl/ctbzuohufuencyodbshtzwyjpmbaw', 'data/euo/pwbvwlaovlstrcheybfkjsjyhdbtq', 'data/evf/waaqpqvzrhmfvpdrsqqwwxqwyzitr', 'data/fzx/ynsvujayvwrbrybvqisdjteghhelm', 'data/hco/aigehntwfivvtdrkcihlqaijmimpq', 'data/hjg/smprjbusqkwpplocvzfblyqbrqmdi', 'data/hjl/qqcrekshhvxwwxsticvpqcbbxhqdp', 'data/htw/tvdbubwgquyarlkmaycdtjvkufopo', 'data/jkq/zvyyptbaqwvdehffvqitvnzoivqov', 'data/kuf/gruwwkmhwuxwyxhdatjjyqtsmxpwb', 'data/mml/lnedccunjsosjiwmngneyywziclwt', 'data/nfd/ryldrcydnezxszamqcuffozixsxqd', 'data/qne/rekqaifotbljgklrnthaewfpxcifo', 'data/sfr/kfkvuhzulqyqnlpgyxgyoiizzbdsj', 'data/uzl/olvwhefyxlmsgzcmeyovnylaabfsp'] run container_id:roottestrecoverytimemetric-gw1-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrecoverytimemetric-gw1-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:10 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Stderr: Container roottestrelativefilepath-gw3-node-1 Stopping Stderr: Container roottestrelativefilepath-gw3-node-1 Stopped Stderr: Container roottestrelativefilepath-gw3-node-1 Removing Stderr: Container roottestrelativefilepath-gw3-node-1 Removed Stderr: Network roottestrelativefilepath-gw3_default Removing Stderr: Network roottestrelativefilepath-gw3_default Removed Cleanup called Docker networks for project roottestrelativefilepath-gw3 are NETWORK ID NAME DRIVER SCOPE run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Docker containers for project roottestrelativefilepath-gw3 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Docker volumes for project roottestrelativefilepath-gw3 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrelativefilepath-gw3-.*-1$' --format '{{.ID}}:{{.Names}}'] Stdout:4157 Unstopped containers: {} No running containers for project: roottestrelativefilepath-gw3 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:5 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 5 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestreplicacanbecomeleader-gw9-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.1.3, port:2181, use_ssl:False Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestreplicacanbecomeleader-gw9-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.1.4, port:2181, use_ssl:False Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/.env --project-name roottestreplicacanbecomeleader-gw9 --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node3/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/.env --project-name roottestreplicacanbecomeleader-gw9 --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node3/docker-compose.yml up -d --no-recreate] http://172.16.5.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=data%2F HTTP/1.1" 200 0 list_objects (18): ['data/dom/yusxjhjmhxyceahdmqqocrbmgytgb', 'data/dsm/abzesqkqilfkwoiuqfqykqczzwmkb', 'data/eox/rdlzgcwqyvhwlhwspdwqjdyxiigkk', 'data/erl/ctbzuohufuencyodbshtzwyjpmbaw', 'data/euo/pwbvwlaovlstrcheybfkjsjyhdbtq', 'data/evf/waaqpqvzrhmfvpdrsqqwwxqwyzitr', 'data/fzx/ynsvujayvwrbrybvqisdjteghhelm', 'data/hco/aigehntwfivvtdrkcihlqaijmimpq', 'data/hjg/smprjbusqkwpplocvzfblyqbrqmdi', 'data/hjl/qqcrekshhvxwwxsticvpqcbbxhqdp', 'data/htw/tvdbubwgquyarlkmaycdtjvkufopo', 'data/jkq/zvyyptbaqwvdehffvqitvnzoivqov', 'data/kuf/gruwwkmhwuxwyxhdatjjyqtsmxpwb', 'data/mml/lnedccunjsosjiwmngneyywziclwt', 'data/nfd/ryldrcydnezxszamqcuffozixsxqd', 'data/qne/rekqaifotbljgklrnthaewfpxcifo', 'data/sfr/kfkvuhzulqyqnlpgyxgyoiizzbdsj', 'data/uzl/olvwhefyxlmsgzcmeyovnylaabfsp'] Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node1-1 Stopping Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node2-1 Stopping Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node1-1 Stopped Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node2-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/.env --project-name roottestprofilesettingsandconstraintsorder-gw0 --file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_profile_settings_and_constraints_order/_instances-0-gw0/node2/docker-compose.yml down --volumes] Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 run container_id:roottestrecoverytimemetric-gw1-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrecoverytimemetric-gw1-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:10 Stderr: Container roottestreplicacanbecomeleader-gw9-zoo2-1 Running Stderr: Container roottestreplicacanbecomeleader-gw9-zoo3-1 Running Stderr: Container roottestreplicacanbecomeleader-gw9-zoo1-1 Running Stderr: Container roottestreplicacanbecomeleader-gw9-node1-1 Creating Stderr: Container roottestreplicacanbecomeleader-gw9-node2-1 Creating Stderr: Container roottestreplicacanbecomeleader-gw9-node3-1 Creating Stderr: Container roottestreplicacanbecomeleader-gw9-node2-1 Created Stderr: Container roottestreplicacanbecomeleader-gw9-node1-1 Created Stderr: Container roottestreplicacanbecomeleader-gw9-node3-1 Created Stderr: Container roottestreplicacanbecomeleader-gw9-node2-1 Starting Stderr: Container roottestreplicacanbecomeleader-gw9-node1-1 Starting Stderr: Container roottestreplicacanbecomeleader-gw9-node3-1 Starting Stderr: Container roottestreplicacanbecomeleader-gw9-node1-1 Started Stderr: Container roottestreplicacanbecomeleader-gw9-node2-1 Started Stderr: Container roottestreplicacanbecomeleader-gw9-node3-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestreplicacanbecomeleader-gw9-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestreplicacanbecomeleader-gw9-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.1.5... http://localhost:None "GET /v1.46/containers/roottestreplicacanbecomeleader-gw9-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/b8947531ee169c72e354b6315d0ee18dd42a9b44b42f61bf77ed7342c0623721/json HTTP/1.1" 200 None run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Stdout:4157 Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node1-1 Stopping Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node2-1 Stopping Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node2-1 Stopped Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node2-1 Removing Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node1-1 Stopped Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node1-1 Removing Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node2-1 Removed Stderr: Container roottestprofilesettingsandconstraintsorder-gw0-node1-1 Removed Stderr: Network roottestprofilesettingsandconstraintsorder-gw0_default Removing Stderr: Network roottestprofilesettingsandconstraintsorder-gw0_default Removed Cleanup called Docker networks for project roottestprofilesettingsandconstraintsorder-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestprofilesettingsandconstraintsorder-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES http://localhost:None "GET /v1.46/containers/b8947531ee169c72e354b6315d0ee18dd42a9b44b42f61bf77ed7342c0623721/json HTTP/1.1" 200 None Docker volumes for project roottestprofilesettingsandconstraintsorder-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestprofilesettingsandconstraintsorder-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestprofilesettingsandconstraintsorder-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:5 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 5 test_replication_without_zookeeper/test.py::test_startup_without_zookeeper http://localhost:None "GET /v1.46/containers/b8947531ee169c72e354b6315d0ee18dd42a9b44b42f61bf77ed7342c0623721/json HTTP/1.1" 200 None Running tests in /ClickHouse/tests/integration/test_replication_without_zookeeper/test.py Cluster start called. is_up=False Docker networks for project roottestreplicationwithoutzookeeper-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicationwithoutzookeeper-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreplicationwithoutzookeeper-gw0 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestreplicationwithoutzookeeper-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicationwithoutzookeeper-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreplicationwithoutzookeeper-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicationwithoutzookeeper-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] http://localhost:None "GET /v1.46/containers/b8947531ee169c72e354b6315d0ee18dd42a9b44b42f61bf77ed7342c0623721/json HTTP/1.1" 200 None Unstopped containers: {} No running containers for project: roottestreplicationwithoutzookeeper-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 Stopping Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 Stopping Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 Stopped Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-config/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-config/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-params/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-params/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Command:[docker compose --env-file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/.env --project-name roottestrenderlogfilenametemplates-gw2 --file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-config/docker-compose.yml --file /ClickHouse/tests/integration/test_render_log_file_name_templates/_instances-0-gw2/file-names-from-params/docker-compose.yml down --volumes] Stdout:5 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 5 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_replication_without_zookeeper/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/node1/database Setup logs dir /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/node1/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /v1.46/containers/b8947531ee169c72e354b6315d0ee18dd42a9b44b42f61bf77ed7342c0623721/json HTTP/1.1" 200 None http://localhost:None "GET /version HTTP/1.1" 200 826 http://172.16.5.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=data%2F HTTP/1.1" 200 0 Command:[docker compose --env-file /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/.env --project-name roottestreplicationwithoutzookeeper-gw0 --file /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml pull] list_objects (18): ['data/dom/yusxjhjmhxyceahdmqqocrbmgytgb', 'data/dsm/abzesqkqilfkwoiuqfqykqczzwmkb', 'data/eox/rdlzgcwqyvhwlhwspdwqjdyxiigkk', 'data/erl/ctbzuohufuencyodbshtzwyjpmbaw', 'data/euo/pwbvwlaovlstrcheybfkjsjyhdbtq', 'data/evf/waaqpqvzrhmfvpdrsqqwwxqwyzitr', 'data/fzx/ynsvujayvwrbrybvqisdjteghhelm', 'data/hco/aigehntwfivvtdrkcihlqaijmimpq', 'data/hjg/smprjbusqkwpplocvzfblyqbrqmdi', 'data/hjl/qqcrekshhvxwwxsticvpqcbbxhqdp', 'data/htw/tvdbubwgquyarlkmaycdtjvkufopo', 'data/jkq/zvyyptbaqwvdehffvqitvnzoivqov', 'data/kuf/gruwwkmhwuxwyxhdatjjyqtsmxpwb', 'data/mml/lnedccunjsosjiwmngneyywziclwt', 'data/nfd/ryldrcydnezxszamqcuffozixsxqd', 'data/qne/rekqaifotbljgklrnthaewfpxcifo', 'data/sfr/kfkvuhzulqyqnlpgyxgyoiizzbdsj', 'data/uzl/olvwhefyxlmsgzcmeyovnylaabfsp'] http://localhost:None "GET /v1.46/containers/b8947531ee169c72e354b6315d0ee18dd42a9b44b42f61bf77ed7342c0623721/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestreplicacanbecomeleader-gw9-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestreplicacanbecomeleader-gw9-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.1.6... http://localhost:None "GET /v1.46/containers/roottestreplicacanbecomeleader-gw9-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/78ec1a72675fc77881bb90f8ca98834eaee3d5441a37b683c59e170e27a932a4/json HTTP/1.1" 200 None ClickHouse node2 started get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestreplicacanbecomeleader-gw9-node3-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestreplicacanbecomeleader-gw9-node3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node3, ip: 172.16.1.7... http://localhost:None "GET /v1.46/containers/roottestreplicacanbecomeleader-gw9-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/67938e105310fec7f910722091a2b44c205412abb3e69f479c413a4c9768ef9f/json HTTP/1.1" 200 None ClickHouse node3 started Executing query CREATE TABLE test_table(date Date, id UInt32, dummy UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_table', '0') PARTITION BY date ORDER BY id on node1 Executing query CREATE TABLE test_table(date Date, id UInt32, dummy UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_table', '1') PARTITION BY date ORDER BY id on node2 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 run container_id:roottestrecoverytimemetric-gw1-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrecoverytimemetric-gw1-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestrecoverytimemetric-gw1-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrecoverytimemetric-gw1-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT default_compression_codec FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node1 Executing query CREATE TABLE test_table(date Date, id UInt32, dummy UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_table', '3') PARTITION BY date ORDER BY id SETTINGS replicated_can_become_leader=0sad on node3 No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrecoverytimemetric-gw1-node-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/7946c790c4c62b345714513c81c458a09ef5753d59f915108b4547f2e5dbcc1b/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/7946c790c4c62b345714513c81c458a09ef5753d59f915108b4547f2e5dbcc1b/json HTTP/1.1" 200 586 Executing query SELECT name FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node2 Executing query select can_become_leader from system.replicas where table = 'test_table' on node1 Executing query SELECT default_compression_codec FROM system.parts where name = 'all_0_0_1' and table = 'recompression_replicated' on node2 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 Stopping Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 Stopping Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 Stopped Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 Removing Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 Stopped Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 Removing Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-params-1 Removed Stderr: Container roottestrenderlogfilenametemplates-gw2-file-names-from-config-1 Removed Stderr: Network roottestrenderlogfilenametemplates-gw2_default Removing Stderr: Network roottestrenderlogfilenametemplates-gw2_default Removed Cleanup called Stdout:4157 Docker networks for project roottestrenderlogfilenametemplates-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrenderlogfilenametemplates-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query select can_become_leader from system.replicas where table = 'test_table' on node2 Docker volumes for project roottestrenderlogfilenametemplates-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrenderlogfilenametemplates-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestrenderlogfilenametemplates-gw2 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:5 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 5 Command:[docker compose --env-file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/.env --project-name roottestreplicacanbecomeleader-gw9 --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node3/docker-compose.yml stop --timeout 20] [gw9] PASSED test_replica_can_become_leader/test.py::test_can_become_leader test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable Running tests in /ClickHouse/tests/integration/test_runtime_configurable_cache_size/test.py Cluster start called. is_up=False [gw6] PASSED test_recompression_ttl/test.py::test_recompression_replicated Executing query CREATE TABLE table_for_recompression (d DateTime, key UInt64, data String) ENGINE MergeTree() ORDER BY tuple() TTL d + INTERVAL 10 SECOND RECOMPRESS CODEC(ZSTD(10)) SETTINGS merge_with_recompression_ttl_timeout = 0 on node1 test_recompression_ttl/test.py::test_recompression_simple Docker networks for project roottestruntimeconfigurablecachesize-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestruntimeconfigurablecachesize-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Stderr: Container roottests3accessheaders-gw4-resolver-1 Stopping Stderr: Container roottests3accessheaders-gw4-node1-1 Stopping Stderr: Container roottests3accessheaders-gw4-node1-1 Stopped Stderr: Container roottests3accessheaders-gw4-minio1-1 Stopping Stderr: Container roottests3accessheaders-gw4-minio1-1 Stopped Stderr: Container roottests3accessheaders-gw4-resolver-1 Stopped Stderr: Container roottests3accessheaders-gw4-proxy2-1 Stopping Stderr: Container roottests3accessheaders-gw4-proxy1-1 Stopping Stderr: Container roottests3accessheaders-gw4-proxy1-1 Stopped Stderr: Container roottests3accessheaders-gw4-proxy2-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Docker volumes for project roottestruntimeconfigurablecachesize-gw2 are DRIVER VOLUME NAME Cleanup called Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/.env --project-name roottests3accessheaders-gw4 --file /ClickHouse/tests/integration/test_s3_access_headers/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml down --volumes] Executing query INSERT INTO table_for_recompression VALUES (now(), 1, '1') on node1 Docker networks for project roottestruntimeconfigurablecachesize-gw2 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestruntimeconfigurablecachesize-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestruntimeconfigurablecachesize-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestruntimeconfigurablecachesize-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestruntimeconfigurablecachesize-gw2 Trying to prune unused networks... Executing query SELECT default_compression_codec FROM system.parts where name = 'all_1_1_0' on node1 Trying to prune unused images... Command:[docker image prune -f] http://172.16.5.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=data%2F HTTP/1.1" 200 0 list_objects (18): ['data/dom/yusxjhjmhxyceahdmqqocrbmgytgb', 'data/dsm/abzesqkqilfkwoiuqfqykqczzwmkb', 'data/eox/rdlzgcwqyvhwlhwspdwqjdyxiigkk', 'data/erl/ctbzuohufuencyodbshtzwyjpmbaw', 'data/euo/pwbvwlaovlstrcheybfkjsjyhdbtq', 'data/evf/waaqpqvzrhmfvpdrsqqwwxqwyzitr', 'data/fzx/ynsvujayvwrbrybvqisdjteghhelm', 'data/hco/aigehntwfivvtdrkcihlqaijmimpq', 'data/hjg/smprjbusqkwpplocvzfblyqbrqmdi', 'data/hjl/qqcrekshhvxwwxsticvpqcbbxhqdp', 'data/htw/tvdbubwgquyarlkmaycdtjvkufopo', 'data/jkq/zvyyptbaqwvdehffvqitvnzoivqov', 'data/kuf/gruwwkmhwuxwyxhdatjjyqtsmxpwb', 'data/mml/lnedccunjsosjiwmngneyywziclwt', 'data/nfd/ryldrcydnezxszamqcuffozixsxqd', 'data/qne/rekqaifotbljgklrnthaewfpxcifo', 'data/sfr/kfkvuhzulqyqnlpgyxgyoiizzbdsj', 'data/uzl/olvwhefyxlmsgzcmeyovnylaabfsp'] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:3 Command:[docker volume prune -f] Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 Stdout:Total reclaimed space: 0B Volumes pruned: 3 Setup directory for instance: node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_runtime_configurable_cache_size/configs/default.xml'] to /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/node/database Setup logs dir /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/node/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/.env --project-name roottestruntimeconfigurablecachesize-gw2 --file /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/node/docker-compose.yml pull] run container_id:roottestrecoverytimemetric-gw1-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrecoverytimemetric-gw1-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:798 Clickhouse process running. run container_id:roottestrecoverytimemetric-gw1-node-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrecoverytimemetric-gw1-node-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:798 Executing query select 20 on node Stderr: Container roottests3accessheaders-gw4-resolver-1 Stopping Stderr: Container roottests3accessheaders-gw4-node1-1 Stopping Stderr: Container roottests3accessheaders-gw4-resolver-1 Stopped Stderr: Container roottests3accessheaders-gw4-resolver-1 Removing Stderr: Container roottests3accessheaders-gw4-node1-1 Stopped Stderr: Container roottests3accessheaders-gw4-node1-1 Removing Stderr: Container roottests3accessheaders-gw4-resolver-1 Removed Stderr: Container roottests3accessheaders-gw4-node1-1 Removed Stderr: Container roottests3accessheaders-gw4-minio1-1 Stopping Stderr: Container roottests3accessheaders-gw4-minio1-1 Stopped Stderr: Container roottests3accessheaders-gw4-minio1-1 Removing Stderr: Container roottests3accessheaders-gw4-minio1-1 Removed Stderr: Container roottests3accessheaders-gw4-proxy2-1 Stopping Stderr: Container roottests3accessheaders-gw4-proxy1-1 Stopping Stderr: Container roottests3accessheaders-gw4-proxy1-1 Stopped Stderr: Container roottests3accessheaders-gw4-proxy1-1 Removing Stderr: Container roottests3accessheaders-gw4-proxy2-1 Stopped Stderr: Container roottests3accessheaders-gw4-proxy2-1 Removing Stderr: Container roottests3accessheaders-gw4-proxy2-1 Removed Stderr: Container roottests3accessheaders-gw4-proxy1-1 Removed Stderr: Volume roottests3accessheaders-gw4_data1-1 Removing Stderr: Network roottests3accessheaders-gw4_default Removing Stderr: Volume roottests3accessheaders-gw4_data1-1 Removed Stderr: Network roottests3accessheaders-gw4_default Removed Cleanup called Docker networks for project roottests3accessheaders-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottests3accessheaders-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottests3accessheaders-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottests3accessheaders-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottests3accessheaders-gw4 Trying to prune unused networks... run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 Stdout:4157 Stdout:3 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 3 test_replica_is_active/test.py::test_replica_is_active Running tests in /ClickHouse/tests/integration/test_replica_is_active/test.py Cluster start called. is_up=False Docker networks for project roottestreplicaisactive-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicaisactive-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreplicaisactive-gw4 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestreplicaisactive-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicaisactive-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreplicaisactive-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicaisactive-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreplicaisactive-gw4 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:3 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 3 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node1/database Setup logs dir /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node2/database Setup logs dir /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node3 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node3/configs/config.d Setup database dir /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node3/database Setup logs dir /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node3/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/.env --project-name roottestreplicaisactive-gw4 --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node3/docker-compose.yml pull] http://172.16.5.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=data%2F HTTP/1.1" 200 0 list_objects (18): ['data/dom/yusxjhjmhxyceahdmqqocrbmgytgb', 'data/dsm/abzesqkqilfkwoiuqfqykqczzwmkb', 'data/eox/rdlzgcwqyvhwlhwspdwqjdyxiigkk', 'data/erl/ctbzuohufuencyodbshtzwyjpmbaw', 'data/euo/pwbvwlaovlstrcheybfkjsjyhdbtq', 'data/evf/waaqpqvzrhmfvpdrsqqwwxqwyzitr', 'data/fzx/ynsvujayvwrbrybvqisdjteghhelm', 'data/hco/aigehntwfivvtdrkcihlqaijmimpq', 'data/hjg/smprjbusqkwpplocvzfblyqbrqmdi', 'data/hjl/qqcrekshhvxwwxsticvpqcbbxhqdp', 'data/htw/tvdbubwgquyarlkmaycdtjvkufopo', 'data/jkq/zvyyptbaqwvdehffvqitvnzoivqov', 'data/kuf/gruwwkmhwuxwyxhdatjjyqtsmxpwb', 'data/mml/lnedccunjsosjiwmngneyywziclwt', 'data/nfd/ryldrcydnezxszamqcuffozixsxqd', 'data/qne/rekqaifotbljgklrnthaewfpxcifo', 'data/sfr/kfkvuhzulqyqnlpgyxgyoiizzbdsj', 'data/uzl/olvwhefyxlmsgzcmeyovnylaabfsp'] Executing query select 20 on node Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 Executing query select 20 on node run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:4157 Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 http://172.16.5.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=data%2F HTTP/1.1" 200 0 list_objects (18): ['data/dom/yusxjhjmhxyceahdmqqocrbmgytgb', 'data/dsm/abzesqkqilfkwoiuqfqykqczzwmkb', 'data/eox/rdlzgcwqyvhwlhwspdwqjdyxiigkk', 'data/erl/ctbzuohufuencyodbshtzwyjpmbaw', 'data/euo/pwbvwlaovlstrcheybfkjsjyhdbtq', 'data/evf/waaqpqvzrhmfvpdrsqqwwxqwyzitr', 'data/fzx/ynsvujayvwrbrybvqisdjteghhelm', 'data/hco/aigehntwfivvtdrkcihlqaijmimpq', 'data/hjg/smprjbusqkwpplocvzfblyqbrqmdi', 'data/hjl/qqcrekshhvxwwxsticvpqcbbxhqdp', 'data/htw/tvdbubwgquyarlkmaycdtjvkufopo', 'data/jkq/zvyyptbaqwvdehffvqitvnzoivqov', 'data/kuf/gruwwkmhwuxwyxhdatjjyqtsmxpwb', 'data/mml/lnedccunjsosjiwmngneyywziclwt', 'data/nfd/ryldrcydnezxszamqcuffozixsxqd', 'data/qne/rekqaifotbljgklrnthaewfpxcifo', 'data/sfr/kfkvuhzulqyqnlpgyxgyoiizzbdsj', 'data/uzl/olvwhefyxlmsgzcmeyovnylaabfsp'] Executing query select 20 on node Executing query SELECT recovery_time FROM system.clusters WHERE cluster = 'rdb' on node Executing query DROP DATABASE rdb on node Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:4157 http://172.16.5.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=data%2F HTTP/1.1" 200 0 list_objects (2): ['data/htw/tvdbubwgquyarlkmaycdtjvkufopo', 'data/nfd/ryldrcydnezxszamqcuffozixsxqd'] http://172.16.5.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=data%2F HTTP/1.1" 200 0 list_objects (2): ['data/htw/tvdbubwgquyarlkmaycdtjvkufopo', 'data/nfd/ryldrcydnezxszamqcuffozixsxqd'] Executing query DROP TABLE IF EXISTS test_hardlinks_preserved_when_projection_dropped SYNC on node1 Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 Executing query DROP TABLE IF EXISTS test_hardlinks_preserved_when_projection_dropped SYNC on node2 Command:[docker compose --env-file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/.env --project-name roottestreplicatedzerocopyprojectionmutation-gw8 --file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node2/docker-compose.yml stop --timeout 20] [gw8] PASSED test_replicated_zero_copy_projection_mutation/test.py::test_hardlinks_preserved_when_projection_dropped Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:4157 Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 Stderr: Container roottestreplicacanbecomeleader-gw9-node3-1 Stopping Stderr: Container roottestreplicacanbecomeleader-gw9-node1-1 Stopping Stderr: Container roottestreplicacanbecomeleader-gw9-node2-1 Stopping Stderr: Container roottestreplicacanbecomeleader-gw9-node3-1 Stopped Stderr: Container roottestreplicacanbecomeleader-gw9-node2-1 Stopped Stderr: Container roottestreplicacanbecomeleader-gw9-node1-1 Stopped Stderr: Container roottestreplicacanbecomeleader-gw9-zoo1-1 Stopping Stderr: Container roottestreplicacanbecomeleader-gw9-zoo2-1 Stopping Stderr: Container roottestreplicacanbecomeleader-gw9-zoo3-1 Stopping Stderr: Container roottestreplicacanbecomeleader-gw9-zoo3-1 Stopped Stderr: Container roottestreplicacanbecomeleader-gw9-zoo2-1 Stopped Stderr: Container roottestreplicacanbecomeleader-gw9-zoo1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/.env --project-name roottestreplicacanbecomeleader-gw9 --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replica_can_become_leader/_instances-0-gw9/node3/docker-compose.yml down --volumes] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:4157 Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 Stderr: Container roottestreplicacanbecomeleader-gw9-node1-1 Stopping Stderr: Container roottestreplicacanbecomeleader-gw9-node3-1 Stopping Stderr: Container roottestreplicacanbecomeleader-gw9-node2-1 Stopping Stderr: Container roottestreplicacanbecomeleader-gw9-node1-1 Stopped Stderr: Container roottestreplicacanbecomeleader-gw9-node1-1 Removing Stderr: Container roottestreplicacanbecomeleader-gw9-node2-1 Stopped Stderr: Container roottestreplicacanbecomeleader-gw9-node2-1 Removing Stderr: Container roottestreplicacanbecomeleader-gw9-node3-1 Stopped Stderr: Container roottestreplicacanbecomeleader-gw9-node3-1 Removing Stderr: Container roottestreplicacanbecomeleader-gw9-node1-1 Removed Stderr: Container roottestreplicacanbecomeleader-gw9-node3-1 Removed Stderr: Container roottestreplicacanbecomeleader-gw9-node2-1 Removed Stderr: Container roottestreplicacanbecomeleader-gw9-zoo2-1 Stopping Stderr: Container roottestreplicacanbecomeleader-gw9-zoo3-1 Stopping Stderr: Container roottestreplicacanbecomeleader-gw9-zoo1-1 Stopping Stderr: Container roottestreplicacanbecomeleader-gw9-zoo2-1 Stopped Stderr: Container roottestreplicacanbecomeleader-gw9-zoo2-1 Removing Stderr: Container roottestreplicacanbecomeleader-gw9-zoo3-1 Stopped Stderr: Container roottestreplicacanbecomeleader-gw9-zoo3-1 Removing Stderr: Container roottestreplicacanbecomeleader-gw9-zoo1-1 Stopped Stderr: Container roottestreplicacanbecomeleader-gw9-zoo1-1 Removing Stderr: Container roottestreplicacanbecomeleader-gw9-zoo2-1 Removed Stderr: Container roottestreplicacanbecomeleader-gw9-zoo3-1 Removed Stderr: Container roottestreplicacanbecomeleader-gw9-zoo1-1 Removed Stderr: Network roottestreplicacanbecomeleader-gw9_default Removing Stderr: Network roottestreplicacanbecomeleader-gw9_default Removed Cleanup called Docker networks for project roottestreplicacanbecomeleader-gw9 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicacanbecomeleader-gw9 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreplicacanbecomeleader-gw9 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicacanbecomeleader-gw9-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreplicacanbecomeleader-gw9 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:3 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 3 Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:4157 Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Command:[docker compose --env-file /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/.env --project-name roottestrecoverytimemetric-gw1 --file /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/node/docker-compose.yml stop --timeout 20] [gw1] PASSED test_recovery_time_metric/test.py::test_recovery_time_metric http://localhost:None "GET /v1.46/exec/31b807f35f1177487aaee6679e959c84f2702eaeba60faa1e2c9faa064913cb3/json HTTP/1.1" 200 584 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrefreshablemv-gw5-node1-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/e15bf79f49d02c9a46ce722094b48118f1e085b3b3e678f57296e77b2a65724a/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/e15bf79f49d02c9a46ce722094b48118f1e085b3b3e678f57296e77b2a65724a/json HTTP/1.1" 200 586 Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 Stderr: Container roottestrecoverytimemetric-gw1-node-1 Stopping Stderr: Container roottestrecoverytimemetric-gw1-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/.env --project-name roottestrecoverytimemetric-gw1 --file /ClickHouse/tests/integration/test_recovery_time_metric/_instances-0-gw1/node/docker-compose.yml down --volumes] Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 Stderr: Container roottestrecoverytimemetric-gw1-node-1 Stopping Stderr: Container roottestrecoverytimemetric-gw1-node-1 Stopped Stderr: Container roottestrecoverytimemetric-gw1-node-1 Removing Stderr: Container roottestrecoverytimemetric-gw1-node-1 Removed Stderr: Network roottestrecoverytimemetric-gw1_default Removing Stderr: Network roottestrecoverytimemetric-gw1_default Removed Cleanup called Docker networks for project roottestrecoverytimemetric-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrecoverytimemetric-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrecoverytimemetric-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrecoverytimemetric-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestrecoverytimemetric-gw1 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:3 Command:[docker volume prune -f] Stdout:5048 Clickhouse process running. run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:Total reclaimed space: 0B Volumes pruned: 3 Running tests in /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/test.py Cluster start called. is_up=False test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers Docker networks for project roottestreloadauxiliaryzookeepers-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreloadauxiliaryzookeepers-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Stdout:5048 Executing query select 20 on node1 Docker volumes for project roottestreloadauxiliaryzookeepers-gw1 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestreloadauxiliaryzookeepers-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreloadauxiliaryzookeepers-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Executing query system refresh view re.a0 on node1 Docker volumes for project roottestreloadauxiliaryzookeepers-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreloadauxiliaryzookeepers-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreloadauxiliaryzookeepers-gw1 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c ps -C clickhouse] Stdout:3 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 3 Setup directory for instance: node Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/node/configs/config.d Setup database dir /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/node/database Setup logs dir /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/node/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/.env --project-name roottestreloadauxiliaryzookeepers-gw1 --file /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml pull] Stdout: PID TTY TIME CMD Stdout: 5048 ? 00:00:01 clickhouse run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c pkill clickhouse] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 Stdout:5048 Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Stderr: node Pulling Stderr: node Pulled ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/.env --project-name roottestruntimeconfigurablecachesize-gw2 --file /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/node/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/.env --project-name roottestruntimeconfigurablecachesize-gw2 --file /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/node/docker-compose.yml up -d --no-recreate] Stderr: zoo1 Skipped - Image is already being pulled by node Stderr: zoo2 Skipped - Image is already being pulled by node Stderr: zoo3 Skipped - Image is already being pulled by node Stderr: node Pulling Stderr: node Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper1/log', '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper1/config', '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper1/coordination', '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper2/log', '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper2/config', '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper2/coordination', '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper3/log', '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper3/config', '/ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/keeper3/coordination'] Command:[docker compose --project-name roottestreloadauxiliaryzookeepers-gw1 --env-file /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5048 Stderr: node1 Skipped - Image is already being pulled by zoo3 Stderr: zoo1 Skipped - Image is already being pulled by zoo3 Stderr: zoo2 Skipped - Image is already being pulled by zoo3 Stderr: zoo3 Pulling Stderr: zoo3 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper1/log', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper1/config', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper1/coordination', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper2/log', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper2/config', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper2/coordination', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper3/log', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper3/config', '/ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/keeper3/coordination'] Command:[docker compose --project-name roottestreplicationwithoutzookeeper-gw0 --env-file /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr: node2 Skipped - Image is already being pulled by zoo3 Stderr: node3 Skipped - Image is already being pulled by zoo3 Stderr: node1 Skipped - Image is already being pulled by zoo3 Stderr: zoo1 Skipped - Image is already being pulled by zoo3 Stderr: zoo2 Skipped - Image is already being pulled by zoo3 Stderr: zoo3 Pulling Stderr: zoo3 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper1/log', '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper1/config', '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper1/coordination', '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper2/log', '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper2/config', '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper2/coordination', '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper3/log', '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper3/config', '/ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/keeper3/coordination'] Command:[docker compose --project-name roottestreplicaisactive-gw4 --env-file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 Connection dropped: outstanding heartbeat ping not received Transition to CONNECTING Connection dropped: socket connection error: Connection refused Stderr: Network roottestruntimeconfigurablecachesize-gw2_default Creating Stderr: Network roottestruntimeconfigurablecachesize-gw2_default Created Stderr: Container roottestruntimeconfigurablecachesize-gw2-node-1 Creating Stderr: Container roottestruntimeconfigurablecachesize-gw2-node-1 Created Stderr: Container roottestruntimeconfigurablecachesize-gw2-node-1 Starting Stderr: Container roottestruntimeconfigurablecachesize-gw2-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestruntimeconfigurablecachesize-gw2-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestruntimeconfigurablecachesize-gw2-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.3.2... http://localhost:None "GET /v1.46/containers/roottestruntimeconfigurablecachesize-gw2-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/746e7084b6d482c9ac4dcf7d7918886288bbb86c93b31cff1255917d2a397a87/json HTTP/1.1" 200 None Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/746e7084b6d482c9ac4dcf7d7918886288bbb86c93b31cff1255917d2a397a87/json HTTP/1.1" 200 None Executing query SELECT name FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 http://localhost:None "GET /v1.46/containers/746e7084b6d482c9ac4dcf7d7918886288bbb86c93b31cff1255917d2a397a87/json HTTP/1.1" 200 None Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/746e7084b6d482c9ac4dcf7d7918886288bbb86c93b31cff1255917d2a397a87/json HTTP/1.1" 200 None Connection dropped: outstanding heartbeat ping not received Transition to CONNECTING Stderr:time="2025-04-02T03:59:36Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestreloadauxiliaryzookeepers-gw1_default Creating Stderr: Network roottestreloadauxiliaryzookeepers-gw1_default Created Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo2-1 Creating Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo3-1 Creating Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo1-1 Creating Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo1-1 Created Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo2-1 Created Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo3-1 Created Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo2-1 Starting Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo3-1 Starting Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo1-1 Starting Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo1-1 Started Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo2-1 Started Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo3-1 Started Stderr:time="2025-04-02T03:59:37Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:59:37Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestreloadauxiliaryzookeepers-gw1-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.1.4, port:2181, use_ssl:False Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/746e7084b6d482c9ac4dcf7d7918886288bbb86c93b31cff1255917d2a397a87/json HTTP/1.1" 200 None Executing query SELECT default_compression_codec FROM system.parts where name = 'all_1_1_1' and table = 'table_for_recompression' on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/746e7084b6d482c9ac4dcf7d7918886288bbb86c93b31cff1255917d2a397a87/json HTTP/1.1" 200 None ClickHouse node started Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SYSTEM DROP QUERY CACHE on node Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 Stdout:5048 Executing query SELECT 1 SETTINGS use_query_cache = 1, query_cache_ttl = 1 on node Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Stderr:time="2025-04-02T03:59:36Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestreplicationwithoutzookeeper-gw0_default Creating Stderr: Network roottestreplicationwithoutzookeeper-gw0_default Created Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo3-1 Creating Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo1-1 Creating Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo2-1 Creating Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo1-1 Created Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo3-1 Created Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo2-1 Created Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo1-1 Starting Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo2-1 Starting Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo3-1 Starting Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo2-1 Started Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo1-1 Started Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo3-1 Started Stderr:time="2025-04-02T03:59:38Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:59:38Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestreplicationwithoutzookeeper-gw0-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.4.4, port:2181, use_ssl:False Connecting to 172.16.4.4(172.16.4.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Stderr:time="2025-04-02T03:59:36Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestreplicaisactive-gw4_default Creating Stderr: Network roottestreplicaisactive-gw4_default Created Stderr: Container roottestreplicaisactive-gw4-zoo3-1 Creating Stderr: Container roottestreplicaisactive-gw4-zoo1-1 Creating Stderr: Container roottestreplicaisactive-gw4-zoo2-1 Creating Stderr: Container roottestreplicaisactive-gw4-zoo1-1 Created Stderr: Container roottestreplicaisactive-gw4-zoo3-1 Created Stderr: Container roottestreplicaisactive-gw4-zoo2-1 Created Stderr: Container roottestreplicaisactive-gw4-zoo3-1 Starting Stderr: Container roottestreplicaisactive-gw4-zoo1-1 Starting Stderr: Container roottestreplicaisactive-gw4-zoo2-1 Starting Stderr: Container roottestreplicaisactive-gw4-zoo3-1 Started Stderr: Container roottestreplicaisactive-gw4-zoo2-1 Started Stderr: Container roottestreplicaisactive-gw4-zoo1-1 Started Stderr:time="2025-04-02T03:59:38Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T03:59:38Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestreplicaisactive-gw4-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.7.2, port:2181, use_ssl:False Connecting to 172.16.7.2(172.16.7.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.4.4(172.16.4.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.7.2(172.16.7.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.4.4(172.16.4.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.7.2(172.16.7.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5048 Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.4.4(172.16.4.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 Connecting to 172.16.7.2(172.16.7.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.4.4(172.16.4.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5048 Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 Executing query SELECT count(*) FROM system.query_cache on node run container_id:roottestruntimeconfigurablecachesize-gw2-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/default.xml) && echo PGNsaWNraG91c2U+CgogICAgPHF1ZXJ5X2NhY2hlPgogICAgICAgIDxtYXhfZW50cmllcz4wPC9tYXhfZW50cmllcz4KICAgIDwvcXVlcnlfY2FjaGU+Cgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/default.xml'] Command:[docker exec roottestruntimeconfigurablecachesize-gw2-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/config.d/default.xml) && echo PGNsaWNraG91c2U+CgogICAgPHF1ZXJ5X2NhY2hlPgogICAgICAgIDxtYXhfZW50cmllcz4wPC9tYXhfZW50cmllcz4KICAgIDwvcXVlcnlfY2FjaGU+Cgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/config.d/default.xml] Executing query SYSTEM RELOAD CONFIG on node Executing query SELECT count(*) FROM system.query_cache on node Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 Connection dropped: socket connection error: Connection refused Executing query SELECT 2 SETTINGS use_query_cache = 1, query_cache_ttl = 1 on node Connecting to 172.16.7.2(172.16.7.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query SELECT count(*) FROM system.query_cache on node Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestreloadauxiliaryzookeepers-gw1-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.1.2, port:2181, use_ssl:False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query SELECT 3 SETTINGS use_query_cache = 1, query_cache_ttl = 1 on node Connection dropped: outstanding heartbeat ping not received Transition to CONNECTING Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestreloadauxiliaryzookeepers-gw1-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.1.3, port:2181, use_ssl:False Executing query SELECT count(*) FROM system.query_cache on node Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/.env --project-name roottestreloadauxiliaryzookeepers-gw1 --file /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/.env --project-name roottestreloadauxiliaryzookeepers-gw1 --file /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate] Stdout:5048 run container_id:roottestruntimeconfigurablecachesize-gw2-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname /etc/clickhouse-server/config.d/default.xml) && echo PGNsaWNraG91c2U+CgogICAgPHF1ZXJ5X2NhY2hlPgogICAgICAgIDxtYXhfZW50cmllcz4yPC9tYXhfZW50cmllcz4KICAgIDwvcXVlcnlfY2FjaGU+CgogICAgPG1hcmtfY2FjaGVfc2l6ZT40OTY8L21hcmtfY2FjaGVfc2l6ZT4KCjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/config.d/default.xml'] Command:[docker exec roottestruntimeconfigurablecachesize-gw2-node-1 bash -c mkdir -p $(dirname /etc/clickhouse-server/config.d/default.xml) && echo PGNsaWNraG91c2U+CgogICAgPHF1ZXJ5X2NhY2hlPgogICAgICAgIDxtYXhfZW50cmllcz4yPC9tYXhfZW50cmllcz4KICAgIDwvcXVlcnlfY2FjaGU+CgogICAgPG1hcmtfY2FjaGVfc2l6ZT40OTY8L21hcmtfY2FjaGVfc2l6ZT4KCjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/config.d/default.xml] Executing query SYSTEM RELOAD CONFIG on node Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 Executing query SELECT 4 SETTINGS use_query_cache = 1, query_cache_ttl = 1 on node Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo1-1 Running Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo2-1 Running Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo3-1 Running Stderr: Container roottestreloadauxiliaryzookeepers-gw1-node-1 Creating Stderr: Container roottestreloadauxiliaryzookeepers-gw1-node-1 Created Stderr: Container roottestreloadauxiliaryzookeepers-gw1-node-1 Starting Stderr: Container roottestreloadauxiliaryzookeepers-gw1-node-1 Started ClickHouse instance created get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestreloadauxiliaryzookeepers-gw1-node-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node http://localhost:None "GET /v1.46/containers/roottestreloadauxiliaryzookeepers-gw1-node-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node, ip: 172.16.1.5... Executing query SELECT count(*) FROM system.query_cache on node http://localhost:None "GET /v1.46/containers/roottestreloadauxiliaryzookeepers-gw1-node-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/611d952a505a5cb5de252093adf9b78b45d97a37db2707d72cb02ff0d15e3ebc/json HTTP/1.1" 200 None Command:[docker compose --env-file /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/.env --project-name roottestruntimeconfigurablecachesize-gw2 --file /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/node/docker-compose.yml stop --timeout 20] [gw2] PASSED test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable http://localhost:None "GET /v1.46/containers/611d952a505a5cb5de252093adf9b78b45d97a37db2707d72cb02ff0d15e3ebc/json HTTP/1.1" 200 None Connecting to 172.16.4.4(172.16.4.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost http://localhost:None "GET /v1.46/containers/611d952a505a5cb5de252093adf9b78b45d97a37db2707d72cb02ff0d15e3ebc/json HTTP/1.1" 200 None Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestreplicationwithoutzookeeper-gw0-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.4.2, port:2181, use_ssl:False Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost http://localhost:None "GET /v1.46/containers/611d952a505a5cb5de252093adf9b78b45d97a37db2707d72cb02ff0d15e3ebc/json HTTP/1.1" 200 None Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestreplicationwithoutzookeeper-gw0-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.4.3, port:2181, use_ssl:False Connecting to 172.16.4.3(172.16.4.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 http://localhost:None "GET /v1.46/containers/611d952a505a5cb5de252093adf9b78b45d97a37db2707d72cb02ff0d15e3ebc/json HTTP/1.1" 200 None ClickHouse node started Executing query CREATE TABLE simple (date Date, id UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/0/simple', 'node') ORDER BY tuple() PARTITION BY date; on node Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/.env --project-name roottestreplicationwithoutzookeeper-gw0 --file /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/.env --project-name roottestreplicationwithoutzookeeper-gw0 --file /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml up -d --no-recreate] Executing query INSERT INTO simple VALUES ('2020-08-27', 1) on node Stderr: Container roottestruntimeconfigurablecachesize-gw2-node-1 Stopping Stderr: Container roottestruntimeconfigurablecachesize-gw2-node-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/.env --project-name roottestruntimeconfigurablecachesize-gw2 --file /ClickHouse/tests/integration/test_runtime_configurable_cache_size/_instances-0-gw2/node/docker-compose.yml down --volumes] Executing query CREATE TABLE simple2 (date Date, id UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/1/simple', 'node') ORDER BY tuple() PARTITION BY date; on node run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5048 run container_id:roottestreloadauxiliaryzookeepers-gw1-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'echo \'\n \n \n zoo1\n 2181\n \n \n zoo2\n 2181\n \n \n zoo3\n 2181\n \n 2000\n \n \n \n \n zoo1\n 2181\n \n \n zoo2\n 2181\n \n \n \n\' > /etc/clickhouse-server/conf.d/zookeeper_config.xml'] Command:[docker exec roottestreloadauxiliaryzookeepers-gw1-node-1 bash -c echo ' zoo1 2181 zoo2 2181 zoo3 2181 2000 zoo1 2181 zoo2 2181 ' > /etc/clickhouse-server/conf.d/zookeeper_config.xml] Executing query SYSTEM RELOAD CONFIG on node Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo1-1 Running Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo2-1 Running Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo3-1 Running Stderr: Container roottestreplicationwithoutzookeeper-gw0-node1-1 Creating Stderr: Container roottestreplicationwithoutzookeeper-gw0-node1-1 Created Stderr: Container roottestreplicationwithoutzookeeper-gw0-node1-1 Starting Stderr: Container roottestreplicationwithoutzookeeper-gw0-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestreplicationwithoutzookeeper-gw0-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestreplicationwithoutzookeeper-gw0-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.4.5... http://localhost:None "GET /v1.46/containers/roottestreplicationwithoutzookeeper-gw0-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d340b6c1f260d71fde42ad243520a9e2c24e800c07347d0e1cf04e430fec873a/json HTTP/1.1" 200 None Connecting to 172.16.7.2(172.16.7.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestreplicaisactive-gw4-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.7.4, port:2181, use_ssl:False Connecting to 172.16.7.4(172.16.7.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Stderr: Container roottestruntimeconfigurablecachesize-gw2-node-1 Stopping Stderr: Container roottestruntimeconfigurablecachesize-gw2-node-1 Stopped Stderr: Container roottestruntimeconfigurablecachesize-gw2-node-1 Removing Stderr: Container roottestruntimeconfigurablecachesize-gw2-node-1 Removed Stderr: Network roottestruntimeconfigurablecachesize-gw2_default Removing Stderr: Network roottestruntimeconfigurablecachesize-gw2_default Removed Cleanup called Docker networks for project roottestruntimeconfigurablecachesize-gw2 are NETWORK ID NAME DRIVER SCOPE http://localhost:None "GET /v1.46/containers/d340b6c1f260d71fde42ad243520a9e2c24e800c07347d0e1cf04e430fec873a/json HTTP/1.1" 200 None Docker containers for project roottestruntimeconfigurablecachesize-gw2 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestruntimeconfigurablecachesize-gw2 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestruntimeconfigurablecachesize-gw2-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestruntimeconfigurablecachesize-gw2 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestreplicaisactive-gw4-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.7.3, port:2181, use_ssl:False Connecting to 172.16.7.3(172.16.7.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost http://localhost:None "GET /v1.46/containers/d340b6c1f260d71fde42ad243520a9e2c24e800c07347d0e1cf04e430fec873a/json HTTP/1.1" 200 None Stdout:3 Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Volumes pruned: 3 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/.env --project-name roottestreplicaisactive-gw4 --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node3/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/.env --project-name roottestreplicaisactive-gw4 --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node3/docker-compose.yml up -d --no-recreate] http://localhost:None "GET /v1.46/containers/d340b6c1f260d71fde42ad243520a9e2c24e800c07347d0e1cf04e430fec873a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d340b6c1f260d71fde42ad243520a9e2c24e800c07347d0e1cf04e430fec873a/json HTTP/1.1" 200 None ClickHouse node1 started Executing query CREATE DATABASE test; CREATE TABLE test_table(date Date, id UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated', 'node1') ORDER BY id PARTITION BY toYYYYMM(date); on node1 Executing query INSERT INTO test_table VALUES ('2018-10-01', 1), ('2018-10-02', 2), ('2018-10-03', 3) on node1 Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5048 Executing query SELECT COUNT(*) from test_table on node1 Stderr: Container roottestreplicaisactive-gw4-zoo2-1 Running Stderr: Container roottestreplicaisactive-gw4-zoo1-1 Running Stderr: Container roottestreplicaisactive-gw4-zoo3-1 Running Stderr: Container roottestreplicaisactive-gw4-node2-1 Creating Stderr: Container roottestreplicaisactive-gw4-node3-1 Creating Stderr: Container roottestreplicaisactive-gw4-node1-1 Creating Stderr: Container roottestreplicaisactive-gw4-node2-1 Created Stderr: Container roottestreplicaisactive-gw4-node3-1 Created Stderr: Container roottestreplicaisactive-gw4-node1-1 Created Stderr: Container roottestreplicaisactive-gw4-node2-1 Starting Stderr: Container roottestreplicaisactive-gw4-node3-1 Starting Stderr: Container roottestreplicaisactive-gw4-node1-1 Starting Stderr: Container roottestreplicaisactive-gw4-node3-1 Started Stderr: Container roottestreplicaisactive-gw4-node1-1 Started Stderr: Container roottestreplicaisactive-gw4-node2-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestreplicaisactive-gw4-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestreplicaisactive-gw4-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.7.6... http://localhost:None "GET /v1.46/containers/roottestreplicaisactive-gw4-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/47247b356ea532f6df71f4e36a67cd217ae80fffac5f42bbbc0847af7fe585d2/json HTTP/1.1" 200 None Executing query SELECT is_readonly from system.replicas where table='test_table' on node1 http://localhost:None "GET /v1.46/containers/47247b356ea532f6df71f4e36a67cd217ae80fffac5f42bbbc0847af7fe585d2/json HTTP/1.1" 200 None get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestreplicationwithoutzookeeper-gw0-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.4.4, port:2181, use_ssl:False Connecting to 172.16.4.4(172.16.4.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED run_kazoo_commands_with_retries: zoo1, Sending request(xid=1): GetChildren(path='/clickhouse', watcher=None) Received response(xid=1): ['tables', 'sessions', 'task_queue'] Sending request(xid=2): GetChildren(path='/clickhouse/tables', watcher=None) Received response(xid=2): ['test'] Sending request(xid=3): GetChildren(path='/clickhouse/tables/test', watcher=None) Received response(xid=3): ['replicated'] Sending request(xid=4): GetChildren(path='/clickhouse/tables/test/replicated', watcher=None) Received response(xid=4): ['blocks', 'part_moves_shard', 'async_blocks', 'log', 'replicas', 'alter_partition_version', 'leader_election', 'nonincrement_block_numbers', 'columns', 'pinned_part_uuids', 'block_numbers', 'mutations', 'temp', 'metadata', 'table_shared_id', 'quorum', 'lost_part_count'] Sending request(xid=5): GetChildren(path='/clickhouse/tables/test/replicated/blocks', watcher=None) Received response(xid=5): ['201810_2956868034535131113_3759223844523231509'] Sending request(xid=6): GetChildren(path='/clickhouse/tables/test/replicated/blocks/201810_2956868034535131113_3759223844523231509', watcher=None) Received response(xid=6): [] Sending request(xid=7): Delete(path='/clickhouse/tables/test/replicated/blocks/201810_2956868034535131113_3759223844523231509', version=-1) Received response(xid=7): True Sending request(xid=8): Delete(path='/clickhouse/tables/test/replicated/blocks', version=-1) Received response(xid=8): True Sending request(xid=9): GetChildren(path='/clickhouse/tables/test/replicated/part_moves_shard', watcher=None) Received response(xid=9): [] Sending request(xid=10): Delete(path='/clickhouse/tables/test/replicated/part_moves_shard', version=-1) Received response(xid=10): True Sending request(xid=11): GetChildren(path='/clickhouse/tables/test/replicated/async_blocks', watcher=None) Received response(xid=11): [] Sending request(xid=12): Delete(path='/clickhouse/tables/test/replicated/async_blocks', version=-1) Received response(xid=12): True Sending request(xid=13): GetChildren(path='/clickhouse/tables/test/replicated/log', watcher=None) Received response(xid=13): ['log-0000000000'] Sending request(xid=14): GetChildren(path='/clickhouse/tables/test/replicated/log/log-0000000000', watcher=None) Received response(xid=14): [] Sending request(xid=15): Delete(path='/clickhouse/tables/test/replicated/log/log-0000000000', version=-1) Received response(xid=15): True Sending request(xid=16): Delete(path='/clickhouse/tables/test/replicated/log', version=-1) Received response(xid=16): True Sending request(xid=17): GetChildren(path='/clickhouse/tables/test/replicated/replicas', watcher=None) Received response(xid=17): ['node1'] Sending request(xid=18): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1', watcher=None) Received response(xid=18): ['flags', 'is_lost', 'metadata', 'is_active', 'log_pointer', 'parts', 'queue', 'host', 'metadata_version', 'min_unprocessed_insert_time', 'columns', 'mutation_pointer', 'max_processed_insert_time', 'creator_info'] Sending request(xid=19): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/flags', watcher=None) Received response(xid=19): [] Sending request(xid=20): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/flags', version=-1) Received response(xid=20): True Sending request(xid=21): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/is_lost', watcher=None) Received response(xid=21): [] Sending request(xid=22): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/is_lost', version=-1) Received response(xid=22): True Sending request(xid=23): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/metadata', watcher=None) Received response(xid=23): [] Sending request(xid=24): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/metadata', version=-1) Received response(xid=24): True Sending request(xid=25): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/is_active', watcher=None) Received response(xid=25): [] Sending request(xid=26): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/is_active', version=-1) Received response(xid=26): True Sending request(xid=27): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/log_pointer', watcher=None) Received response(xid=27): [] Sending request(xid=28): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/log_pointer', version=-1) Received response(xid=28): True Sending request(xid=29): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/parts', watcher=None) Received response(xid=29): ['201810_0_0_0'] Sending request(xid=30): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/parts/201810_0_0_0', watcher=None) Received response(xid=30): [] Sending request(xid=31): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/parts/201810_0_0_0', version=-1) Received response(xid=31): True Sending request(xid=32): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/parts', version=-1) Received response(xid=32): True Sending request(xid=33): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/queue', watcher=None) Received response(xid=33): [] Sending request(xid=34): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/queue', version=-1) Received response(xid=34): True Sending request(xid=35): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/host', watcher=None) Received response(xid=35): [] Sending request(xid=36): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/host', version=-1) Received response(xid=36): True Sending request(xid=37): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/metadata_version', watcher=None) Received response(xid=37): [] Sending request(xid=38): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/metadata_version', version=-1) Received response(xid=38): True Sending request(xid=39): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/min_unprocessed_insert_time', watcher=None) Received response(xid=39): [] Sending request(xid=40): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/min_unprocessed_insert_time', version=-1) Received response(xid=40): True Sending request(xid=41): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/columns', watcher=None) Received response(xid=41): [] Sending request(xid=42): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/columns', version=-1) Received response(xid=42): True Sending request(xid=43): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/mutation_pointer', watcher=None) Received response(xid=43): [] Sending request(xid=44): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/mutation_pointer', version=-1) Received response(xid=44): True Sending request(xid=45): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/max_processed_insert_time', watcher=None) Received response(xid=45): [] Sending request(xid=46): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/max_processed_insert_time', version=-1) Received response(xid=46): True Sending request(xid=47): GetChildren(path='/clickhouse/tables/test/replicated/replicas/node1/creator_info', watcher=None) Received response(xid=47): [] Sending request(xid=48): Delete(path='/clickhouse/tables/test/replicated/replicas/node1/creator_info', version=-1) Received response(xid=48): True Sending request(xid=49): Delete(path='/clickhouse/tables/test/replicated/replicas/node1', version=-1) Received response(xid=49): True Sending request(xid=50): Delete(path='/clickhouse/tables/test/replicated/replicas', version=-1) Received response(xid=50): True Sending request(xid=51): GetChildren(path='/clickhouse/tables/test/replicated/alter_partition_version', watcher=None) Received response(xid=51): [] Sending request(xid=52): Delete(path='/clickhouse/tables/test/replicated/alter_partition_version', version=-1) Received response(xid=52): True Sending request(xid=53): GetChildren(path='/clickhouse/tables/test/replicated/leader_election', watcher=None) Received response(xid=53): ['leader_election-0'] Sending request(xid=54): GetChildren(path='/clickhouse/tables/test/replicated/leader_election/leader_election-0', watcher=None) Received response(xid=54): [] Sending request(xid=55): Delete(path='/clickhouse/tables/test/replicated/leader_election/leader_election-0', version=-1) Received response(xid=55): True Sending request(xid=56): Delete(path='/clickhouse/tables/test/replicated/leader_election', version=-1) Received response(xid=56): True Sending request(xid=57): GetChildren(path='/clickhouse/tables/test/replicated/nonincrement_block_numbers', watcher=None) Received response(xid=57): [] Sending request(xid=58): Delete(path='/clickhouse/tables/test/replicated/nonincrement_block_numbers', version=-1) Received response(xid=58): True Sending request(xid=59): GetChildren(path='/clickhouse/tables/test/replicated/columns', watcher=None) Received response(xid=59): [] Sending request(xid=60): Delete(path='/clickhouse/tables/test/replicated/columns', version=-1) Received response(xid=60): True Sending request(xid=61): GetChildren(path='/clickhouse/tables/test/replicated/pinned_part_uuids', watcher=None) Received response(xid=61): [] Sending request(xid=62): Delete(path='/clickhouse/tables/test/replicated/pinned_part_uuids', version=-1) Received response(xid=62): True Sending request(xid=63): GetChildren(path='/clickhouse/tables/test/replicated/block_numbers', watcher=None) Received response(xid=63): ['201810'] Sending request(xid=64): GetChildren(path='/clickhouse/tables/test/replicated/block_numbers/201810', watcher=None) Received response(xid=64): [] Sending request(xid=65): Delete(path='/clickhouse/tables/test/replicated/block_numbers/201810', version=-1) Received response(xid=65): True Sending request(xid=66): Delete(path='/clickhouse/tables/test/replicated/block_numbers', version=-1) Received response(xid=66): True Sending request(xid=67): GetChildren(path='/clickhouse/tables/test/replicated/mutations', watcher=None) Received response(xid=67): [] Sending request(xid=68): Delete(path='/clickhouse/tables/test/replicated/mutations', version=-1) Received response(xid=68): True Sending request(xid=69): GetChildren(path='/clickhouse/tables/test/replicated/temp', watcher=None) Received response(xid=69): ['abandonable_lock-other', 'abandonable_lock-insert'] Sending request(xid=70): GetChildren(path='/clickhouse/tables/test/replicated/temp/abandonable_lock-other', watcher=None) Received response(xid=70): [] Sending request(xid=71): Delete(path='/clickhouse/tables/test/replicated/temp/abandonable_lock-other', version=-1) Received response(xid=71): True Sending request(xid=72): GetChildren(path='/clickhouse/tables/test/replicated/temp/abandonable_lock-insert', watcher=None) Received response(xid=72): [] Sending request(xid=73): Delete(path='/clickhouse/tables/test/replicated/temp/abandonable_lock-insert', version=-1) Received response(xid=73): True Sending request(xid=74): Delete(path='/clickhouse/tables/test/replicated/temp', version=-1) Received response(xid=74): True Sending request(xid=75): GetChildren(path='/clickhouse/tables/test/replicated/metadata', watcher=None) Received response(xid=75): [] Sending request(xid=76): Delete(path='/clickhouse/tables/test/replicated/metadata', version=-1) Received response(xid=76): True Sending request(xid=77): GetChildren(path='/clickhouse/tables/test/replicated/table_shared_id', watcher=None) Received response(xid=77): [] Sending request(xid=78): Delete(path='/clickhouse/tables/test/replicated/table_shared_id', version=-1) Received response(xid=78): True Sending request(xid=79): GetChildren(path='/clickhouse/tables/test/replicated/quorum', watcher=None) Received response(xid=79): ['failed_parts', 'last_part', 'parallel'] Sending request(xid=80): GetChildren(path='/clickhouse/tables/test/replicated/quorum/failed_parts', watcher=None) Received response(xid=80): [] Sending request(xid=81): Delete(path='/clickhouse/tables/test/replicated/quorum/failed_parts', version=-1) Received response(xid=81): True Sending request(xid=82): GetChildren(path='/clickhouse/tables/test/replicated/quorum/last_part', watcher=None) Received response(xid=82): [] Sending request(xid=83): Delete(path='/clickhouse/tables/test/replicated/quorum/last_part', version=-1) Received response(xid=83): True Sending request(xid=84): GetChildren(path='/clickhouse/tables/test/replicated/quorum/parallel', watcher=None) Received response(xid=84): [] Sending request(xid=85): Delete(path='/clickhouse/tables/test/replicated/quorum/parallel', version=-1) http://localhost:None "GET /v1.46/containers/47247b356ea532f6df71f4e36a67cd217ae80fffac5f42bbbc0847af7fe585d2/json HTTP/1.1" 200 None Received response(xid=85): True Sending request(xid=86): Delete(path='/clickhouse/tables/test/replicated/quorum', version=-1) Received response(xid=86): True Sending request(xid=87): GetChildren(path='/clickhouse/tables/test/replicated/lost_part_count', watcher=None) Received response(xid=87): [] Sending request(xid=88): Delete(path='/clickhouse/tables/test/replicated/lost_part_count', version=-1) Received response(xid=88): True Sending request(xid=89): Delete(path='/clickhouse/tables/test/replicated', version=-1) Received response(xid=89): True Sending request(xid=90): Delete(path='/clickhouse/tables/test', version=-1) Received response(xid=90): True Sending request(xid=91): Delete(path='/clickhouse/tables', version=-1) Received response(xid=91): True Sending request(xid=92): GetChildren(path='/clickhouse/sessions', watcher=None) Received response(xid=92): ['zookeeper'] Sending request(xid=93): GetChildren(path='/clickhouse/sessions/zookeeper', watcher=None) Received response(xid=93): ['b28da2d5-e3d7-4009-b923-cf77a4f76b35'] Sending request(xid=94): GetChildren(path='/clickhouse/sessions/zookeeper/b28da2d5-e3d7-4009-b923-cf77a4f76b35', watcher=None) Received response(xid=94): [] Sending request(xid=95): Delete(path='/clickhouse/sessions/zookeeper/b28da2d5-e3d7-4009-b923-cf77a4f76b35', version=-1) Received response(xid=95): True Sending request(xid=96): Delete(path='/clickhouse/sessions/zookeeper', version=-1) Received response(xid=96): True Sending request(xid=97): Delete(path='/clickhouse/sessions', version=-1) Received response(xid=97): True Sending request(xid=98): GetChildren(path='/clickhouse/task_queue', watcher=None) Received response(xid=98): ['ddl', 'replicas'] Sending request(xid=99): GetChildren(path='/clickhouse/task_queue/ddl', watcher=None) Received response(xid=99): [] Sending request(xid=100): Delete(path='/clickhouse/task_queue/ddl', version=-1) Received response(xid=100): True Sending request(xid=101): GetChildren(path='/clickhouse/task_queue/replicas', watcher=None) Received response(xid=101): ['node1:9000'] Sending request(xid=102): GetChildren(path='/clickhouse/task_queue/replicas/node1:9000', watcher=None) Received response(xid=102): ['active'] Sending request(xid=103): GetChildren(path='/clickhouse/task_queue/replicas/node1:9000/active', watcher=None) Received response(xid=103): [] Sending request(xid=104): Delete(path='/clickhouse/task_queue/replicas/node1:9000/active', version=-1) Received response(xid=104): True Sending request(xid=105): Delete(path='/clickhouse/task_queue/replicas/node1:9000', version=-1) Received response(xid=105): True Sending request(xid=106): Delete(path='/clickhouse/task_queue/replicas', version=-1) Received response(xid=106): True Sending request(xid=107): Delete(path='/clickhouse/task_queue', version=-1) Received response(xid=107): True Sending request(xid=108): Delete(path='/clickhouse', version=-1) Received response(xid=108): True Sending request(xid=109): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost http://localhost:None "GET /v1.46/containers/47247b356ea532f6df71f4e36a67cd217ae80fffac5f42bbbc0847af7fe585d2/json HTTP/1.1" 200 None Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED http://localhost:None "GET /v1.46/containers/47247b356ea532f6df71f4e36a67cd217ae80fffac5f42bbbc0847af7fe585d2/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestreplicaisactive-gw4-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestreplicaisactive-gw4-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.7.7... http://localhost:None "GET /v1.46/containers/roottestreplicaisactive-gw4-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/60c2dbe87368c1b74dfd26f1b085791bba15bb32bf91f0be52d72b79097afff5/json HTTP/1.1" 200 None ClickHouse node2 started get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestreplicaisactive-gw4-node3-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestreplicaisactive-gw4-node3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node3, ip: 172.16.7.5... http://localhost:None "GET /v1.46/containers/roottestreplicaisactive-gw4-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/7aaf500896b6d7bd28e04dccf146bd624165b83a4d2aedb77d48fd6cf1147ad8/json HTTP/1.1" 200 None ClickHouse node3 started Executing query CREATE TABLE test_table(date Date, id UInt32, dummy UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_table', 'node1') PARTITION BY date ORDER BY id on node1 Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 Executing query CREATE TABLE test_table(date Date, id UInt32, dummy UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_table', 'node2') PARTITION BY date ORDER BY id on node2 Executing query CREATE TABLE test_table(date Date, id UInt32, dummy UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_table', 'node3') PARTITION BY date ORDER BY id on node3 Executing query select replica_is_active from system.replicas where table = 'test_table' on node1 http://localhost:None "GET /v1.46/containers/roottestreplicaisactive-gw4-node3-1/json HTTP/1.1" 200 None run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5048 Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5048 Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/exec/e15bf79f49d02c9a46ce722094b48118f1e085b3b3e678f57296e77b2a65724a/json HTTP/1.1" 200 584 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrefreshablemv-gw5-node1-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/5492f4015473564e06d0865fac5c0365880a736ccaa2d92b884f839d6872a826/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/5492f4015473564e06d0865fac5c0365880a736ccaa2d92b884f839d6872a826/json HTTP/1.1" 200 586 Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 Executing query ALTER TABLE simple2 FETCH PARTITION '2020-08-27' FROM 'zookeeper2:/clickhouse/tables/0/simple'; on node Executing query ALTER TABLE simple2 ATTACH PARTITION '2020-08-27'; on node run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5943 Clickhouse process running. run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query SELECT id FROM simple2 on node Stdout:5943 Executing query select 20 on node1 run container_id:roottestreloadauxiliaryzookeepers-gw1-node-1 detach:False nothrow:False cmd: ['bash', '-c', 'echo \'\n \n \n zoo2\n 2181\n \n 2000\n \n\' > /etc/clickhouse-server/conf.d/zookeeper_config.xml'] Command:[docker exec roottestreloadauxiliaryzookeepers-gw1-node-1 bash -c echo ' zoo2 2181 2000 ' > /etc/clickhouse-server/conf.d/zookeeper_config.xml] Executing query SYSTEM RELOAD CONFIG on node Executing query system refresh view re.a1 on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 5943 ? 00:00:01 clickhouse run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c pkill clickhouse] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5943 Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 http://localhost:None "POST /v1.46/containers/7aaf500896b6d7bd28e04dccf146bd624165b83a4d2aedb77d48fd6cf1147ad8/stop HTTP/1.1" 204 0 Executing query select replica_is_active from system.replicas where table = 'test_table' on node1 http://localhost:None "GET /v1.46/containers/roottestreplicaisactive-gw4-node2-1/json HTTP/1.1" 200 None Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 Executing query SELECT COUNT(*) from test_table on node1 Executing query INSERT INTO test_table VALUES ('2018-10-01', 1), ('2018-10-02', 2), ('2018-10-03', 3) on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5943 run container_id:roottestreplicationwithoutzookeeper-gw0-node1-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestreplicationwithoutzookeeper-gw0-node1-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 10 ? 00:00:00 clickhouse run container_id:roottestreplicationwithoutzookeeper-gw0-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestreplicationwithoutzookeeper-gw0-node1-1 bash -c pkill clickhouse] Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node run container_id:roottestreplicationwithoutzookeeper-gw0-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicationwithoutzookeeper-gw0-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:10 Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 Executing query SELECT name FROM system.parts where name = 'all_1_1_2' and table = 'table_for_recompression' on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5943 run container_id:roottestreplicationwithoutzookeeper-gw0-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicationwithoutzookeeper-gw0-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:10 Executing query OPTIMIZE TABLE table_for_recompression FINAL on node1 Executing query SELECT default_compression_codec FROM system.parts where name = 'all_1_1_2' on node1 Command:[docker compose --env-file /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/.env --project-name roottestrecompressionttl-gw6 --file /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node2/docker-compose.yml stop --timeout 20] [gw6] PASSED test_recompression_ttl/test.py::test_recompression_simple run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5943 run container_id:roottestreplicationwithoutzookeeper-gw0-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicationwithoutzookeeper-gw0-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:10 http://localhost:None "POST /v1.46/containers/60c2dbe87368c1b74dfd26f1b085791bba15bb32bf91f0be52d72b79097afff5/stop HTTP/1.1" 204 0 Executing query select replica_is_active from system.replicas where table = 'test_table' on node1 Command:[docker compose --env-file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/.env --project-name roottestreplicaisactive-gw4 --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node3/docker-compose.yml stop --timeout 20] [gw4] PASSED test_replica_is_active/test.py::test_replica_is_active Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo2-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-minio1-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo1-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo3-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-minio1-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo2-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo1-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo3-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy2-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy1-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy1-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy2-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/.env --project-name roottestreplicatedzerocopyprojectionmutation-gw8 --file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_replicated_zero_copy_projection_mutation/_instances-0-gw8/node2/docker-compose.yml down --volumes] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5943 run container_id:roottestreplicationwithoutzookeeper-gw0-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicationwithoutzookeeper-gw0-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestreplicationwithoutzookeeper-gw0-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicationwithoutzookeeper-gw0-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestreplicationwithoutzookeeper-gw0-node1-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/008353b418fbed002884cececb5cf37bc5921bb7283beb45c7bd5853be46d997/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/008353b418fbed002884cececb5cf37bc5921bb7283beb45c7bd5853be46d997/json HTTP/1.1" 200 586 Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 Removing Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 Removing Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 Removing Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node2-1 Removed Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-resolver-1 Removed Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-node1-1 Removed Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo1-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-minio1-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo2-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo3-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-minio1-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-minio1-1 Removing Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo1-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo1-1 Removing Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo2-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo2-1 Removing Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo3-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo3-1 Removing Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo1-1 Removed Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo2-1 Removed Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-minio1-1 Removed Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy1-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy2-1 Stopping Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy2-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy2-1 Removing Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy1-1 Stopped Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy1-1 Removing Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-zoo3-1 Removed Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy2-1 Removed Stderr: Container roottestreplicatedzerocopyprojectionmutation-gw8-proxy1-1 Removed Stderr: Volume roottestreplicatedzerocopyprojectionmutation-gw8_data1-1 Removing Stderr: Network roottestreplicatedzerocopyprojectionmutation-gw8_default Removing Stderr: Volume roottestreplicatedzerocopyprojectionmutation-gw8_data1-1 Removed Stderr: Network roottestreplicatedzerocopyprojectionmutation-gw8_default Removed Cleanup called Docker networks for project roottestreplicatedzerocopyprojectionmutation-gw8 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicatedzerocopyprojectionmutation-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreplicatedzerocopyprojectionmutation-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicatedzerocopyprojectionmutation-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreplicatedzerocopyprojectionmutation-gw8 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_read_only_table/test.py::test_restart_zookeeper Running tests in /ClickHouse/tests/integration/test_read_only_table/test.py Cluster start called. is_up=False Docker networks for project roottestreadonlytable-gw8 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreadonlytable-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreadonlytable-gw8 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestreadonlytable-gw8 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreadonlytable-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreadonlytable-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreadonlytable-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreadonlytable-gw8 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node1/database Setup logs dir /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node2/database Setup logs dir /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node3 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node3/configs/config.d Setup database dir /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node3/database Setup logs dir /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node3/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/.env --project-name roottestreadonlytable-gw8 --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node3/docker-compose.yml pull] Executing query ALTER TABLE simple2 FETCH PARTITION '2020-08-27' FROM 'zookeeper2:/clickhouse/tables/0/simple'; on node Executing query SELECT id FROM simple2 on node Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/.env --project-name roottestreloadauxiliaryzookeepers-gw1 --file /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop --timeout 20] [gw1] PASSED test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5943 run container_id:roottestreplicationwithoutzookeeper-gw0-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicationwithoutzookeeper-gw0-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:769 Clickhouse process running. run container_id:roottestreplicationwithoutzookeeper-gw0-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestreplicationwithoutzookeeper-gw0-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:769 Executing query select 20 on node1 Executing query SELECT COUNT(*) from test_table on node1 Executing query SELECT is_readonly from system.replicas where table='test_table' on node1 Command:[docker compose --env-file /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/.env --project-name roottestreplicationwithoutzookeeper-gw0 --file /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop --timeout 20] [gw0] PASSED test_replication_without_zookeeper/test.py::test_startup_without_zookeeper run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5943 Stderr: Container roottestreplicationwithoutzookeeper-gw0-node1-1 Stopping Stderr: Container roottestreplicationwithoutzookeeper-gw0-node1-1 Stopped Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo2-1 Stopping Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo3-1 Stopping Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo1-1 Stopping Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo1-1 Stopped Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo2-1 Stopped Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo3-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/.env --project-name roottestreplicationwithoutzookeeper-gw0 --file /ClickHouse/tests/integration/test_replication_without_zookeeper/_instances-0-gw0/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml down --volumes] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5943 Stderr: Container roottestrecompressionttl-gw6-node1-1 Stopping Stderr: Container roottestrecompressionttl-gw6-node2-1 Stopping Stderr: Container roottestrecompressionttl-gw6-node2-1 Stopped Stderr: Container roottestrecompressionttl-gw6-node1-1 Stopped Stderr: Container roottestrecompressionttl-gw6-zoo3-1 Stopping Stderr: Container roottestrecompressionttl-gw6-zoo1-1 Stopping Stderr: Container roottestrecompressionttl-gw6-zoo2-1 Stopping Stderr: Container roottestrecompressionttl-gw6-zoo2-1 Stopped Stderr: Container roottestrecompressionttl-gw6-zoo3-1 Stopped Stderr: Container roottestrecompressionttl-gw6-zoo1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/.env --project-name roottestrecompressionttl-gw6 --file /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_recompression_ttl/_instances-0-gw6/node2/docker-compose.yml down --volumes] Stderr: Container roottestreplicationwithoutzookeeper-gw0-node1-1 Stopping Stderr: Container roottestreplicationwithoutzookeeper-gw0-node1-1 Stopped Stderr: Container roottestreplicationwithoutzookeeper-gw0-node1-1 Removing Stderr: Container roottestreplicationwithoutzookeeper-gw0-node1-1 Removed Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo2-1 Stopping Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo3-1 Stopping Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo1-1 Stopping Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo2-1 Stopped Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo2-1 Removing Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo1-1 Stopped Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo1-1 Removing Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo3-1 Stopped Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo3-1 Removing Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo3-1 Removed Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo2-1 Removed Stderr: Container roottestreplicationwithoutzookeeper-gw0-zoo1-1 Removed Stderr: Network roottestreplicationwithoutzookeeper-gw0_default Removing Stderr: Network roottestreplicationwithoutzookeeper-gw0_default Removed Cleanup called Docker networks for project roottestreplicationwithoutzookeeper-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicationwithoutzookeeper-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreplicationwithoutzookeeper-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicationwithoutzookeeper-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreplicationwithoutzookeeper-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:5943 Stderr: Container roottestrecompressionttl-gw6-node2-1 Stopping Stderr: Container roottestrecompressionttl-gw6-node1-1 Stopping Stderr: Container roottestrecompressionttl-gw6-node2-1 Stopped Stderr: Container roottestrecompressionttl-gw6-node2-1 Removing Stderr: Container roottestrecompressionttl-gw6-node1-1 Stopped Stderr: Container roottestrecompressionttl-gw6-node1-1 Removing Stderr: Container roottestrecompressionttl-gw6-node2-1 Removed Stderr: Container roottestrecompressionttl-gw6-node1-1 Removed Stderr: Container roottestrecompressionttl-gw6-zoo1-1 Stopping Stderr: Container roottestrecompressionttl-gw6-zoo2-1 Stopping Stderr: Container roottestrecompressionttl-gw6-zoo3-1 Stopping Stderr: Container roottestrecompressionttl-gw6-zoo3-1 Stopped Stderr: Container roottestrecompressionttl-gw6-zoo3-1 Removing Stderr: Container roottestrecompressionttl-gw6-zoo1-1 Stopped Stderr: Container roottestrecompressionttl-gw6-zoo1-1 Removing Stderr: Container roottestrecompressionttl-gw6-zoo2-1 Stopped Stderr: Container roottestrecompressionttl-gw6-zoo2-1 Removing Stderr: Container roottestrecompressionttl-gw6-zoo2-1 Removed Stderr: Container roottestrecompressionttl-gw6-zoo1-1 Removed Stderr: Container roottestrecompressionttl-gw6-zoo3-1 Removed Stderr: Network roottestrecompressionttl-gw6_default Removing Stderr: Network roottestrecompressionttl-gw6_default Removed Cleanup called Docker networks for project roottestrecompressionttl-gw6 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrecompressionttl-gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrecompressionttl-gw6 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrecompressionttl-gw6-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestrecompressionttl-gw6 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stderr: Container roottestreplicaisactive-gw4-node1-1 Stopping Stderr: Container roottestreplicaisactive-gw4-node2-1 Stopping Stderr: Container roottestreplicaisactive-gw4-node3-1 Stopping Stderr: Container roottestreplicaisactive-gw4-node2-1 Stopped Stderr: Container roottestreplicaisactive-gw4-node3-1 Stopped Stderr: Container roottestreplicaisactive-gw4-node1-1 Stopped Stderr: Container roottestreplicaisactive-gw4-zoo3-1 Stopping Stderr: Container roottestreplicaisactive-gw4-zoo1-1 Stopping Stderr: Container roottestreplicaisactive-gw4-zoo2-1 Stopping Stderr: Container roottestreplicaisactive-gw4-zoo2-1 Stopped Stderr: Container roottestreplicaisactive-gw4-zoo1-1 Stopped Stderr: Container roottestreplicaisactive-gw4-zoo3-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:1 Volumes pruned: 1 Command:[bash -c [ -f /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/.env --project-name roottestreplicaisactive-gw4 --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_replica_is_active/_instances-0-gw4/node3/docker-compose.yml down --volumes] Stderr: Container roottestreplicaisactive-gw4-node1-1 Stopping Stderr: Container roottestreplicaisactive-gw4-node2-1 Stopping Stderr: Container roottestreplicaisactive-gw4-node3-1 Stopping Stderr: Container roottestreplicaisactive-gw4-node2-1 Stopped Stderr: Container roottestreplicaisactive-gw4-node2-1 Removing Stderr: Container roottestreplicaisactive-gw4-node3-1 Stopped Stderr: Container roottestreplicaisactive-gw4-node3-1 Removing Stderr: Container roottestreplicaisactive-gw4-node1-1 Stopped Stderr: Container roottestreplicaisactive-gw4-node1-1 Removing Stderr: Container roottestreplicaisactive-gw4-node1-1 Removed Stderr: Container roottestreplicaisactive-gw4-node2-1 Removed Stderr: Container roottestreplicaisactive-gw4-node3-1 Removed Stderr: Container roottestreplicaisactive-gw4-zoo2-1 Stopping Stderr: Container roottestreplicaisactive-gw4-zoo1-1 Stopping Stderr: Container roottestreplicaisactive-gw4-zoo3-1 Stopping Stderr: Container roottestreplicaisactive-gw4-zoo1-1 Stopped Stderr: Container roottestreplicaisactive-gw4-zoo1-1 Removing Stderr: Container roottestreplicaisactive-gw4-zoo2-1 Stopped Stderr: Container roottestreplicaisactive-gw4-zoo2-1 Removing Stderr: Container roottestreplicaisactive-gw4-zoo3-1 Stopped Stderr: Container roottestreplicaisactive-gw4-zoo3-1 Removing Stderr: Container roottestreplicaisactive-gw4-zoo1-1 Removed Stderr: Container roottestreplicaisactive-gw4-zoo3-1 Removed Stderr: Container roottestreplicaisactive-gw4-zoo2-1 Removed Stderr: Network roottestreplicaisactive-gw4_default Removing Stderr: Network roottestreplicaisactive-gw4_default Removed Cleanup called Docker networks for project roottestreplicaisactive-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicaisactive-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreplicaisactive-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicaisactive-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreplicaisactive-gw4 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 test_replicating_constants/test.py::test_different_versions Running tests in /ClickHouse/tests/integration/test_replicating_constants/test.py Cluster start called. is_up=False Docker networks for project roottestreplicatingconstants-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicatingconstants-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Docker volumes for project roottestreplicatingconstants-gw4 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottestreplicatingconstants-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicatingconstants-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Stdout:5943 Docker volumes for project roottestreplicatingconstants-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicatingconstants-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreplicatingconstants-gw4 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node1/database Setup logs dir /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node2/database Setup logs dir /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node2/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:8b2301119731', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/.env --project-name roottestreplicatingconstants-gw4 --file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node2/docker-compose.yml pull] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/exec/5492f4015473564e06d0865fac5c0365880a736ccaa2d92b884f839d6872a826/json HTTP/1.1" 200 584 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrefreshablemv-gw5-node1-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/131f192d7826c34d4cbe615a72db9853d6cf6afa6fc7f479cbcfa03129960687/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/131f192d7826c34d4cbe615a72db9853d6cf6afa6fc7f479cbcfa03129960687/json HTTP/1.1" 200 586 Stderr: Container roottestreloadauxiliaryzookeepers-gw1-node-1 Stopping Stderr: Container roottestreloadauxiliaryzookeepers-gw1-node-1 Stopped Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo3-1 Stopping Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo1-1 Stopping Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo2-1 Stopping Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo3-1 Stopped Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo1-1 Stopped Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo2-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/.env --project-name roottestreloadauxiliaryzookeepers-gw1 --file /ClickHouse/tests/integration/test_reload_auxiliary_zookeepers/_instances-0-gw1/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml down --volumes] Stderr: Container roottestreloadauxiliaryzookeepers-gw1-node-1 Stopping Stderr: Container roottestreloadauxiliaryzookeepers-gw1-node-1 Stopped Stderr: Container roottestreloadauxiliaryzookeepers-gw1-node-1 Removing Stderr: Container roottestreloadauxiliaryzookeepers-gw1-node-1 Removed Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo1-1 Stopping Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo2-1 Stopping Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo3-1 Stopping Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo1-1 Stopped Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo1-1 Removing Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo3-1 Stopped Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo3-1 Removing Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo2-1 Stopped Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo2-1 Removing Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo1-1 Removed Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo3-1 Removed Stderr: Container roottestreloadauxiliaryzookeepers-gw1-zoo2-1 Removed Stderr: Network roottestreloadauxiliaryzookeepers-gw1_default Removing Stderr: Network roottestreloadauxiliaryzookeepers-gw1_default Removed Cleanup called Docker networks for project roottestreloadauxiliaryzookeepers-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreloadauxiliaryzookeepers-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreloadauxiliaryzookeepers-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreloadauxiliaryzookeepers-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Unstopped containers: {} No running containers for project: roottestreloadauxiliaryzookeepers-gw1 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:6835 Clickhouse process running. run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Stdout:6835 Executing query select 20 on node1 Executing query system refresh view re.a1 on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 6835 ? 00:00:01 clickhouse run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c pkill clickhouse] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:6835 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:6835 Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:6835 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:6835 Stderr: node1 Skipped - Image is already being pulled by zoo2 Stderr: zoo3 Skipped - Image is already being pulled by zoo2 Stderr: zoo1 Skipped - Image is already being pulled by zoo2 Stderr: zoo2 Pulling Stderr: node2 Pulling Stderr: zoo2 Pulled Stderr: node2 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper1/log', '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper1/config', '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper1/coordination', '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper2/log', '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper2/config', '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper2/coordination', '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper3/log', '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper3/config', '/ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/keeper3/coordination'] Command:[docker compose --project-name roottestreplicatingconstants-gw4 --env-file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr: zoo1 Skipped - Image is already being pulled by node1 Stderr: zoo2 Skipped - Image is already being pulled by node1 Stderr: zoo3 Skipped - Image is already being pulled by node1 Stderr: node2 Skipped - Image is already being pulled by node1 Stderr: node3 Skipped - Image is already being pulled by node1 Stderr: node1 Pulling Stderr: node1 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper1/log', '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper1/config', '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper1/coordination', '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper2/log', '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper2/config', '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper2/coordination', '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper3/log', '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper3/config', '/ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/keeper3/coordination'] Command:[docker compose --project-name roottestreadonlytable-gw8 --env-file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:6835 Stderr:time="2025-04-02T04:00:03Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestreplicatingconstants-gw4_default Creating Stderr: Network roottestreplicatingconstants-gw4_default Created Stderr: Container roottestreplicatingconstants-gw4-zoo3-1 Creating Stderr: Container roottestreplicatingconstants-gw4-zoo1-1 Creating Stderr: Container roottestreplicatingconstants-gw4-zoo2-1 Creating Stderr: Container roottestreplicatingconstants-gw4-zoo2-1 Created Stderr: Container roottestreplicatingconstants-gw4-zoo1-1 Created Stderr: Container roottestreplicatingconstants-gw4-zoo3-1 Created Stderr: Container roottestreplicatingconstants-gw4-zoo1-1 Starting Stderr: Container roottestreplicatingconstants-gw4-zoo3-1 Starting Stderr: Container roottestreplicatingconstants-gw4-zoo2-1 Starting Stderr: Container roottestreplicatingconstants-gw4-zoo1-1 Started Stderr: Container roottestreplicatingconstants-gw4-zoo2-1 Started Stderr: Container roottestreplicatingconstants-gw4-zoo3-1 Started Stderr:time="2025-04-02T04:00:03Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T04:00:03Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestreplicatingconstants-gw4-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.1.2, port:2181, use_ssl:False Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Stderr:time="2025-04-02T04:00:03Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestreadonlytable-gw8_default Creating Stderr: Network roottestreadonlytable-gw8_default Created Stderr: Container roottestreadonlytable-gw8-zoo3-1 Creating Stderr: Container roottestreadonlytable-gw8-zoo1-1 Creating Stderr: Container roottestreadonlytable-gw8-zoo2-1 Creating Stderr: Container roottestreadonlytable-gw8-zoo1-1 Created Stderr: Container roottestreadonlytable-gw8-zoo2-1 Created Stderr: Container roottestreadonlytable-gw8-zoo3-1 Created Stderr: Container roottestreadonlytable-gw8-zoo3-1 Starting Stderr: Container roottestreadonlytable-gw8-zoo2-1 Starting Stderr: Container roottestreadonlytable-gw8-zoo1-1 Starting Stderr: Container roottestreadonlytable-gw8-zoo2-1 Started Stderr: Container roottestreadonlytable-gw8-zoo1-1 Started Stderr: Container roottestreadonlytable-gw8-zoo3-1 Started Stderr:time="2025-04-02T04:00:04Z" level=debug msg="otel error" error="" Stderr:time="2025-04-02T04:00:04Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestreadonlytable-gw8-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.3.4, port:2181, use_ssl:False Connecting to 172.16.3.4(172.16.3.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.3.4(172.16.3.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.3.4(172.16.3.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Connecting to 172.16.3.4(172.16.3.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Stdout:6835 Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.3.4(172.16.3.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:6835 Connecting to 172.16.1.2(172.16.1.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestreplicatingconstants-gw4-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.1.4, port:2181, use_ssl:False Connecting to 172.16.1.4(172.16.1.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestreplicatingconstants-gw4-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.1.3, port:2181, use_ssl:False Connecting to 172.16.1.3(172.16.1.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:6835 Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/.env --project-name roottestreplicatingconstants-gw4 --file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node2/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/.env --project-name roottestreplicatingconstants-gw4 --file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node2/docker-compose.yml up -d --no-recreate] Stderr: Container roottestreplicatingconstants-gw4-zoo3-1 Running Stderr: Container roottestreplicatingconstants-gw4-zoo1-1 Running Stderr: Container roottestreplicatingconstants-gw4-zoo2-1 Running Stderr: Container roottestreplicatingconstants-gw4-node2-1 Creating Stderr: Container roottestreplicatingconstants-gw4-node1-1 Creating Stderr: Container roottestreplicatingconstants-gw4-node1-1 Created Stderr: Container roottestreplicatingconstants-gw4-node2-1 Created Stderr: Container roottestreplicatingconstants-gw4-node2-1 Starting Stderr: Container roottestreplicatingconstants-gw4-node1-1 Starting Stderr: Container roottestreplicatingconstants-gw4-node2-1 Started Stderr: Container roottestreplicatingconstants-gw4-node1-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestreplicatingconstants-gw4-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestreplicatingconstants-gw4-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.1.6... http://localhost:None "GET /v1.46/containers/roottestreplicatingconstants-gw4-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/abcfc1fc47d7c0b4b1a59ee9b61daaefc89f4a391da8f36ef0e62ef76d6e1474/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/abcfc1fc47d7c0b4b1a59ee9b61daaefc89f4a391da8f36ef0e62ef76d6e1474/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/abcfc1fc47d7c0b4b1a59ee9b61daaefc89f4a391da8f36ef0e62ef76d6e1474/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/abcfc1fc47d7c0b4b1a59ee9b61daaefc89f4a391da8f36ef0e62ef76d6e1474/json HTTP/1.1" 200 None Connecting to 172.16.3.4(172.16.3.4):2181, use_ssl: False Connection dropped: socket connection error: Connection refused http://localhost:None "GET /v1.46/containers/abcfc1fc47d7c0b4b1a59ee9b61daaefc89f4a391da8f36ef0e62ef76d6e1474/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestreplicatingconstants-gw4-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestreplicatingconstants-gw4-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.1.5... http://localhost:None "GET /v1.46/containers/roottestreplicatingconstants-gw4-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d09283f5909bc3d8aa7b7a96b12686d50ad64eea2330f31e210572df9c6e1d90/json HTTP/1.1" 200 None ClickHouse node2 started Executing query SELECT uniqExact(x) FROM (SELECT version() as x from remote('node{1,2}', system.one)) on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Command:[docker compose --env-file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/.env --project-name roottestreplicatingconstants-gw4 --file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node2/docker-compose.yml stop --timeout 20] [gw4] PASSED test_replicating_constants/test.py::test_different_versions Stdout:6835 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:6835 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:6835 Connection dropped: socket connection error: None run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/exec/131f192d7826c34d4cbe615a72db9853d6cf6afa6fc7f479cbcfa03129960687/json HTTP/1.1" 200 584 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrefreshablemv-gw5-node1-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/11d27d23019024fdeac03a808ac722e26f4356509f5ebf6e4c98b226c8b46bee/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/11d27d23019024fdeac03a808ac722e26f4356509f5ebf6e4c98b226c8b46bee/json HTTP/1.1" 200 586 Connection dropped: socket connection error: None run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:7742 Clickhouse process running. run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:7742 Executing query select 20 on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 7742 ? 00:00:01 clickhouse run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c pkill clickhouse] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:7742 Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:7742 Connecting to 172.16.3.4(172.16.3.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestreadonlytable-gw8-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.3.2, port:2181, use_ssl:False Connecting to 172.16.3.2(172.16.3.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestreadonlytable-gw8-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.3.3, port:2181, use_ssl:False Connecting to 172.16.3.3(172.16.3.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/.env --project-name roottestreadonlytable-gw8 --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node3/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/.env --project-name roottestreadonlytable-gw8 --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node3/docker-compose.yml up -d --no-recreate] Connection dropped: socket connection error: None run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stderr: Container roottestreadonlytable-gw8-zoo3-1 Running Stderr: Container roottestreadonlytable-gw8-zoo1-1 Running Stderr: Container roottestreadonlytable-gw8-zoo2-1 Running Stderr: Container roottestreadonlytable-gw8-node2-1 Creating Stderr: Container roottestreadonlytable-gw8-node1-1 Creating Stderr: Container roottestreadonlytable-gw8-node3-1 Creating Stderr: Container roottestreadonlytable-gw8-node2-1 Created Stderr: Container roottestreadonlytable-gw8-node1-1 Created Stderr: Container roottestreadonlytable-gw8-node3-1 Created Stderr: Container roottestreadonlytable-gw8-node3-1 Starting Stderr: Container roottestreadonlytable-gw8-node2-1 Starting Stderr: Container roottestreadonlytable-gw8-node1-1 Starting Stderr: Container roottestreadonlytable-gw8-node1-1 Started Stderr: Container roottestreadonlytable-gw8-node2-1 Started Stderr: Container roottestreadonlytable-gw8-node3-1 Started ClickHouse instance created get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestreadonlytable-gw8-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestreadonlytable-gw8-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.3.5... http://localhost:None "GET /v1.46/containers/roottestreadonlytable-gw8-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d2d84f2e873afdc10b72a82bd23b2a6abd2735faa10af28c0b8a8091af819313/json HTTP/1.1" 200 None Stdout:7742 http://localhost:None "GET /v1.46/containers/d2d84f2e873afdc10b72a82bd23b2a6abd2735faa10af28c0b8a8091af819313/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d2d84f2e873afdc10b72a82bd23b2a6abd2735faa10af28c0b8a8091af819313/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d2d84f2e873afdc10b72a82bd23b2a6abd2735faa10af28c0b8a8091af819313/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/d2d84f2e873afdc10b72a82bd23b2a6abd2735faa10af28c0b8a8091af819313/json HTTP/1.1" 200 None ClickHouse node1 started get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestreadonlytable-gw8-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestreadonlytable-gw8-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.3.7... http://localhost:None "GET /v1.46/containers/roottestreadonlytable-gw8-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/63d333eff0978e210c075360dd5e130320fbd63a761086f7e98dfb207772b85d/json HTTP/1.1" 200 None ClickHouse node2 started get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestreadonlytable-gw8-node3-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestreadonlytable-gw8-node3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node3, ip: 172.16.3.6... http://localhost:None "GET /v1.46/containers/roottestreadonlytable-gw8-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e71cd1862553ded6b02f9587791e19f08e29d9cda666b2a49414b9973cc435f6/json HTTP/1.1" 200 None ClickHouse node3 started Executing query CREATE TABLE test_table_0(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/0', 'node1') ORDER BY tuple(); on node1 Executing query CREATE TABLE test_table_0(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/0', 'node2') ORDER BY tuple(); on node2 Executing query CREATE TABLE test_table_0(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/0', 'node3') ORDER BY tuple(); on node3 Executing query CREATE TABLE test_table_1(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/1', 'node1') ORDER BY tuple(); on node1 Executing query CREATE TABLE test_table_1(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/1', 'node2') ORDER BY tuple(); on node2 Executing query CREATE TABLE test_table_1(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/1', 'node3') ORDER BY tuple(); on node3 Executing query CREATE TABLE test_table_2(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/2', 'node1') ORDER BY tuple(); on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:7742 Executing query CREATE TABLE test_table_2(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/2', 'node2') ORDER BY tuple(); on node2 Executing query CREATE TABLE test_table_2(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/2', 'node3') ORDER BY tuple(); on node3 Executing query CREATE TABLE test_table_3(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/3', 'node1') ORDER BY tuple(); on node1 Executing query CREATE TABLE test_table_3(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/3', 'node2') ORDER BY tuple(); on node2 Executing query CREATE TABLE test_table_3(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/3', 'node3') ORDER BY tuple(); on node3 Executing query CREATE TABLE test_table_4(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/4', 'node1') ORDER BY tuple(); on node1 Executing query CREATE TABLE test_table_4(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/4', 'node2') ORDER BY tuple(); on node2 Executing query CREATE TABLE test_table_4(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/4', 'node3') ORDER BY tuple(); on node3 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:7742 Executing query CREATE TABLE test_table_5(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/5', 'node1') ORDER BY tuple(); on node1 Executing query CREATE TABLE test_table_5(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/5', 'node2') ORDER BY tuple(); on node2 Executing query CREATE TABLE test_table_5(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/5', 'node3') ORDER BY tuple(); on node3 Executing query CREATE TABLE test_table_6(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/6', 'node1') ORDER BY tuple(); on node1 Executing query CREATE TABLE test_table_6(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/6', 'node2') ORDER BY tuple(); on node2 Executing query CREATE TABLE test_table_6(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/6', 'node3') ORDER BY tuple(); on node3 Executing query CREATE TABLE test_table_7(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/7', 'node1') ORDER BY tuple(); on node1 Executing query CREATE TABLE test_table_7(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/7', 'node2') ORDER BY tuple(); on node2 Executing query CREATE TABLE test_table_7(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/7', 'node3') ORDER BY tuple(); on node3 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:7742 Executing query CREATE TABLE test_table_8(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/8', 'node1') ORDER BY tuple(); on node1 Executing query CREATE TABLE test_table_8(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/8', 'node2') ORDER BY tuple(); on node2 Executing query CREATE TABLE test_table_8(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/8', 'node3') ORDER BY tuple(); on node3 Executing query CREATE TABLE test_table_9(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/9', 'node1') ORDER BY tuple(); on node1 Executing query CREATE TABLE test_table_9(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/9', 'node2') ORDER BY tuple(); on node2 Executing query CREATE TABLE test_table_9(a UInt64) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test/replicated/9', 'node3') ORDER BY tuple(); on node3 Executing query INSERT INTO test_table_0 VALUES (1), (2), (3), (4), (5); on node1 Executing query INSERT INTO test_table_1 VALUES (1), (2), (3), (4), (5); on node1 Executing query INSERT INTO test_table_2 VALUES (1), (2), (3), (4), (5); on node1 Executing query INSERT INTO test_table_3 VALUES (1), (2), (3), (4), (5); on node1 Executing query INSERT INTO test_table_4 VALUES (1), (2), (3), (4), (5); on node1 Executing query INSERT INTO test_table_5 VALUES (1), (2), (3), (4), (5); on node1 Executing query INSERT INTO test_table_6 VALUES (1), (2), (3), (4), (5); on node1 Executing query INSERT INTO test_table_7 VALUES (1), (2), (3), (4), (5); on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query INSERT INTO test_table_8 VALUES (1), (2), (3), (4), (5); on node1 Stdout:7742 Executing query INSERT INTO test_table_9 VALUES (1), (2), (3), (4), (5); on node1 Inserted test data and initialized all tables run container_id:roottestreadonlytable-gw8-node1-1 detach:False nothrow:False cmd: ['ss', '--resolve', '--tcp', '--no-header', 'state', 'ESTABLISHED', '( dport = 2181 or sport = 2181 )'] Command:[docker exec -u root --privileged roottestreadonlytable-gw8-node1-1 ss --resolve --tcp --no-header state ESTABLISHED ( dport = 2181 or sport = 2181 )] Stdout:0 0 node1:52750 roottestreadonlytable-gw8-zoo2-1.roottestreadonlytable-gw8_default:2181 Stopping zookeeper node: zoo2 Command:[docker compose --project-name roottestreadonlytable-gw8 --env-file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop zoo2] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:7742 Stderr: Container roottestreplicatingconstants-gw4-node2-1 Stopping Stderr: Container roottestreplicatingconstants-gw4-node1-1 Stopping Stderr: Container roottestreplicatingconstants-gw4-node2-1 Stopped Stderr: Container roottestreplicatingconstants-gw4-node1-1 Stopped Stderr: Container roottestreplicatingconstants-gw4-zoo1-1 Stopping Stderr: Container roottestreplicatingconstants-gw4-zoo2-1 Stopping Stderr: Container roottestreplicatingconstants-gw4-zoo3-1 Stopping Stderr: Container roottestreplicatingconstants-gw4-zoo2-1 Stopped Stderr: Container roottestreplicatingconstants-gw4-zoo1-1 Stopped Stderr: Container roottestreplicatingconstants-gw4-zoo3-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Command:[docker compose --env-file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/.env --project-name roottestreplicatingconstants-gw4 --file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_replicating_constants/_instances-0-gw4/node2/docker-compose.yml down --volumes] Stdout:7742 Stderr: Container roottestreplicatingconstants-gw4-node2-1 Stopping Stderr: Container roottestreplicatingconstants-gw4-node1-1 Stopping Stderr: Container roottestreplicatingconstants-gw4-node1-1 Stopped Stderr: Container roottestreplicatingconstants-gw4-node1-1 Removing Stderr: Container roottestreplicatingconstants-gw4-node2-1 Stopped Stderr: Container roottestreplicatingconstants-gw4-node2-1 Removing Stderr: Container roottestreplicatingconstants-gw4-node2-1 Removed Stderr: Container roottestreplicatingconstants-gw4-node1-1 Removed Stderr: Container roottestreplicatingconstants-gw4-zoo2-1 Stopping Stderr: Container roottestreplicatingconstants-gw4-zoo1-1 Stopping Stderr: Container roottestreplicatingconstants-gw4-zoo3-1 Stopping Stderr: Container roottestreplicatingconstants-gw4-zoo1-1 Stopped Stderr: Container roottestreplicatingconstants-gw4-zoo1-1 Removing Stderr: Container roottestreplicatingconstants-gw4-zoo2-1 Stopped Stderr: Container roottestreplicatingconstants-gw4-zoo2-1 Removing Stderr: Container roottestreplicatingconstants-gw4-zoo3-1 Stopped Stderr: Container roottestreplicatingconstants-gw4-zoo3-1 Removing Stderr: Container roottestreplicatingconstants-gw4-zoo3-1 Removed Stderr: Container roottestreplicatingconstants-gw4-zoo2-1 Removed Stderr: Container roottestreplicatingconstants-gw4-zoo1-1 Removed Stderr: Network roottestreplicatingconstants-gw4_default Removing Stderr: Network roottestreplicatingconstants-gw4_default Removed Cleanup called Docker networks for project roottestreplicatingconstants-gw4 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreplicatingconstants-gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreplicatingconstants-gw4 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreplicatingconstants-gw4-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreplicatingconstants-gw4 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:7742 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:7742 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/exec/11d27d23019024fdeac03a808ac722e26f4356509f5ebf6e4c98b226c8b46bee/json HTTP/1.1" 200 584 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrefreshablemv-gw5-node1-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/59419067e544de10a3bcfa2fccff6a9114b29d108f6626513a669ecedafc7190/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/59419067e544de10a3bcfa2fccff6a9114b29d108f6626513a669ecedafc7190/json HTTP/1.1" 200 587 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8647 Clickhouse process running. run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stderr: Container roottestreadonlytable-gw8-zoo2-1 Stopping Stderr: Container roottestreadonlytable-gw8-zoo2-1 Stopped Stdout:8647 Executing query select 20 on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 8647 ? 00:00:01 clickhouse run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c pkill clickhouse] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8647 Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8647 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8647 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8647 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8647 Executing query INSERT INTO test_table_0 VALUES (6), (7), (8), (9), (10); on node1 Executing query INSERT INTO test_table_1 VALUES (6), (7), (8), (9), (10); on node1 Executing query INSERT INTO test_table_2 VALUES (6), (7), (8), (9), (10); on node1 Executing query INSERT INTO test_table_3 VALUES (6), (7), (8), (9), (10); on node1 Executing query INSERT INTO test_table_4 VALUES (6), (7), (8), (9), (10); on node1 Executing query INSERT INTO test_table_5 VALUES (6), (7), (8), (9), (10); on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Executing query INSERT INTO test_table_6 VALUES (6), (7), (8), (9), (10); on node1 Stdout:8647 Executing query INSERT INTO test_table_7 VALUES (6), (7), (8), (9), (10); on node1 Executing query INSERT INTO test_table_8 VALUES (6), (7), (8), (9), (10); on node1 Executing query INSERT INTO test_table_9 VALUES (6), (7), (8), (9), (10); on node1 Starting zookeeper node: zoo2 Command:[docker compose --project-name roottestreadonlytable-gw8 --env-file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml start zoo2] Stderr: Container roottestreadonlytable-gw8-zoo2-1 Starting Stderr: Container roottestreadonlytable-gw8-zoo2-1 Started Command:[docker compose --env-file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/.env --project-name roottestreadonlytable-gw8 --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node3/docker-compose.yml stop --timeout 20] [gw8] PASSED test_read_only_table/test.py::test_restart_zookeeper run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8647 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8647 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8647 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:8647 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] http://localhost:None "GET /v1.46/exec/59419067e544de10a3bcfa2fccff6a9114b29d108f6626513a669ecedafc7190/json HTTP/1.1" 200 585 Command:[bash -c [ -f /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs/clickhouse-server.log ] && zgrep -aH "view refreshes failed to stop" /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs/clickhouse-server.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs/clickhouse-server.log ] && zgrep -aH "Closed connections. But" /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs/clickhouse-server.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs/clickhouse-server.log ] && zgrep -aH "Will shutdown forcefully." /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs/clickhouse-server.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs/clickhouse-server.log ] && zgrep -aH "##########" /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs/clickhouse-server.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs/clickhouse-server.log ] && zgrep -aH "===test_refresh_vs_shutdown_smoke start===" /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs/clickhouse-server.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Stdout:/ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs/clickhouse-server.log.8.gz:2025.04.02 03:58:46.118329 [ 775 ] {d434c0ab-6b9e-428c-9ba9-0519ff900b14} executeQuery: (from 172.16.2.1:55756) (query 1, line 1) select '===test_refresh_vs_shutdown_smoke start===' (stage: Complete) Stdout:/ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs/clickhouse-server.log.8.gz:2025.04.02 03:58:46.118348 [ 775 ] {d434c0ab-6b9e-428c-9ba9-0519ff900b14} CancellationChecker: Did not add the task because the timeout is 0. Query: select '===test_refresh_vs_shutdown_smoke start===' run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrefreshablemv-gw5-node1-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/a881cc8f39aff31d9a99ebf6240f0c2cb4da043ad9355665b3162e45d7ee3374/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/a881cc8f39aff31d9a99ebf6240f0c2cb4da043ad9355665b3162e45d7ee3374/json HTTP/1.1" 200 587 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:9538 Clickhouse process running. run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:9538 Executing query select 20 on node1 Executing query drop database re sync on node1 Executing query drop database re sync on node2 Stderr: Container roottestreadonlytable-gw8-node1-1 Stopping Stderr: Container roottestreadonlytable-gw8-node2-1 Stopping Stderr: Container roottestreadonlytable-gw8-node3-1 Stopping Stderr: Container roottestreadonlytable-gw8-node3-1 Stopped Stderr: Container roottestreadonlytable-gw8-node2-1 Stopped Stderr: Container roottestreadonlytable-gw8-node1-1 Stopped Stderr: Container roottestreadonlytable-gw8-zoo3-1 Stopping Stderr: Container roottestreadonlytable-gw8-zoo1-1 Stopping Stderr: Container roottestreadonlytable-gw8-zoo2-1 Stopping Stderr: Container roottestreadonlytable-gw8-zoo3-1 Stopped Stderr: Container roottestreadonlytable-gw8-zoo2-1 Stopped Stderr: Container roottestreadonlytable-gw8-zoo1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node Command:[bash -c [ -f /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/.env --project-name roottestreadonlytable-gw8 --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_read_only_table/_instances-0-gw8/node3/docker-compose.yml down --volumes] Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster2' and host_name='node_1' on node run container_id:roottestreloadclustersconfig-gw7-node-1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n\n \n \n \n true\n \n node_1\n 9000\n \n \n node_2\n 9000\n \n \n \n \n \n true\n \n node_1\n 9000\n \n \n node_2\n 9000\n \n \n \n \n \n true\n \n node_1\n 9000\n \n \n \n \n\n' > /etc/clickhouse-server/config.d/remote_servers.xml"] Command:[docker exec roottestreloadclustersconfig-gw7-node-1 bash -c echo ' true node_1 9000 node_2 9000 true node_1 9000 node_2 9000 true node_1 9000 ' > /etc/clickhouse-server/config.d/remote_servers.xml] Executing query SYSTEM RELOAD CONFIG on node Stderr: Container roottestreadonlytable-gw8-node3-1 Stopping Stderr: Container roottestreadonlytable-gw8-node1-1 Stopping Stderr: Container roottestreadonlytable-gw8-node2-1 Stopping Stderr: Container roottestreadonlytable-gw8-node3-1 Stopped Stderr: Container roottestreadonlytable-gw8-node3-1 Removing Stderr: Container roottestreadonlytable-gw8-node2-1 Stopped Stderr: Container roottestreadonlytable-gw8-node2-1 Removing Stderr: Container roottestreadonlytable-gw8-node1-1 Stopped Stderr: Container roottestreadonlytable-gw8-node1-1 Removing Stderr: Container roottestreadonlytable-gw8-node1-1 Removed Stderr: Container roottestreadonlytable-gw8-node3-1 Removed Stderr: Container roottestreadonlytable-gw8-node2-1 Removed Stderr: Container roottestreadonlytable-gw8-zoo3-1 Stopping Stderr: Container roottestreadonlytable-gw8-zoo1-1 Stopping Stderr: Container roottestreadonlytable-gw8-zoo2-1 Stopping Stderr: Container roottestreadonlytable-gw8-zoo3-1 Stopped Stderr: Container roottestreadonlytable-gw8-zoo3-1 Removing Stderr: Container roottestreadonlytable-gw8-zoo1-1 Stopped Stderr: Container roottestreadonlytable-gw8-zoo1-1 Removing Stderr: Container roottestreadonlytable-gw8-zoo2-1 Stopped Stderr: Container roottestreadonlytable-gw8-zoo2-1 Removing Stderr: Container roottestreadonlytable-gw8-zoo3-1 Removed Stderr: Container roottestreadonlytable-gw8-zoo2-1 Removed Stderr: Container roottestreadonlytable-gw8-zoo1-1 Removed Stderr: Network roottestreadonlytable-gw8_default Removing Stderr: Network roottestreadonlytable-gw8_default Removed Cleanup called Docker networks for project roottestreadonlytable-gw8 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreadonlytable-gw8 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreadonlytable-gw8 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreadonlytable-gw8-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreadonlytable-gw8 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 [gw5] PASSED test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node1 Executing query create database re engine = Replicated('/test/re', 'shard1', '{replica}'); on node2 Executing query create materialized view re.a refresh every 1 second (x Int64) engine Memory as select 1 as x on node1 Executing query create materialized view re.a refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select number*10 as x from numbers(2) on node1 Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster2' and host_name='node_1' on node Executing query SELECT * FROM system.clusters WHERE cluster='test_cluster3' on node Executing query system sync database replica re on node1 Executing query system wait view re.a on node1 Executing query select * from re.a order by all on node1 Executing query select database, view, last_success_time != 0, last_refresh_time != 0, last_refresh_replica in ('1','2'), exception from system.view_refreshes on node1 Executing query system wait view re.a on node2 Executing query select * from re.a order by all on node2 Executing query select database, view, last_success_time != 0, last_refresh_time != 0, last_refresh_replica in ('1','2'), exception from system.view_refreshes on node2 Executing query create materialized view re.append refresh every 1 year append (x Int64) engine ReplicatedMergeTree order by x as select rand() as x on node2 Executing query system test view re.append set fake time '2040-01-01 00:00:01' on node1 Executing query system test view re.append set fake time '2040-01-01 00:00:01' on node2 Connection dropped: socket connection error: None Executing query system wait view re.append; system refresh view re.append; system wait view re.append; on node1 Executing query system wait view re.append; system refresh view re.append; system wait view re.append; on node2 Executing query select count() from re.append on node1 Executing query system test view re.append set fake time '2041-01-01 00:00:01' on node1 Executing query system test view re.append set fake time '2041-01-01 00:00:01' on node2 Executing query select status, last_success_time from system.view_refreshes where view = 'append' on node1 Executing query select status, last_success_time from system.view_refreshes where view = 'append' on node1 Executing query system wait view re.append on node1 Executing query select status, last_success_time from system.view_refreshes where view = 'append' on node2 Executing query system wait view re.append on node2 Executing query system sync replica re.append on node2 Executing query select count() from re.append on node2 Executing query create materialized view re.append_uncoordinated refresh every 1 year settings all_replicas = 1 append (x Int64) engine ReplicatedMergeTree order by x as select rand() as x on node2 Executing query system test view re.append_uncoordinated set fake time '2040-01-01 00:00:01' on node1 Executing query system test view re.append_uncoordinated set fake time '2040-01-01 00:00:01' on node2 Executing query system wait view re.append_uncoordinated; system refresh view re.append_uncoordinated; system wait view re.append_uncoordinated; on node1 Executing query system wait view re.append_uncoordinated; system refresh view re.append_uncoordinated; system wait view re.append_uncoordinated; on node2 Executing query select count() from re.append_uncoordinated on node1 Executing query system test view re.append_uncoordinated set fake time '2041-01-01 00:00:01' on node1 Executing query system test view re.append_uncoordinated set fake time '2041-01-01 00:00:01' on node2 Executing query select status, last_success_time from system.view_refreshes where view = 'append_uncoordinated' on node1 Executing query select status, last_success_time from system.view_refreshes where view = 'append_uncoordinated' on node1 Executing query system wait view re.append_uncoordinated on node1 Executing query select status, last_success_time from system.view_refreshes where view = 'append_uncoordinated' on node2 Executing query system wait view re.append_uncoordinated on node2 Executing query system sync replica re.append_uncoordinated on node2 Executing query select count() from re.append_uncoordinated on node2 Executing query create materialized view re.unreplicated_uncoordinated refresh every 1 second settings all_replicas = 1 append (x String) engine Memory as select 1 as x on node1 Executing query system sync database replica re on node2 Executing query system wait view re.unreplicated_uncoordinated on node1 Executing query select distinct x from re.unreplicated_uncoordinated on node1 Executing query system wait view re.unreplicated_uncoordinated on node2 Executing query select distinct x from re.unreplicated_uncoordinated on node2 Executing query create materialized view re.c refresh every 1 year (x Int64) engine ReplicatedMergeTree order by x empty as select rand() as x on node2 Executing query system sync database replica re on node1 Executing query rename table re.c to re.d on node1 Executing query alter table re.d modify query select number + sleepEachRow(1) as x from numbers(5) settings max_block_size = 1 on node1 Executing query system refresh view re.d on node1 Executing query select status from system.view_refreshes where view = 'd' on node2 Executing query rename table re.d to re.e on node2 Executing query system wait view re.e on node1 Connection dropped: socket connection error: None Executing query select * from re.e order by x on node1 Executing query create materialized view re.f refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select sleepEachRow(1) as x from numbers(1000000) settings max_block_size = 1 on node1 run container_id:roottestreloadclustersconfig-gw7-node-1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n\n \n \n \n true\n \n node_1\n 9000\n \n \n node_2\n 9000\n \n \n \n \n \n true\n \n node_1\n 9000\n \n \n node_2\n 9000\n \n \n \n \n\n' > /etc/clickhouse-server/config.d/remote_servers.xml"] Command:[docker exec roottestreloadclustersconfig-gw7-node-1 bash -c echo ' true node_1 9000 node_2 9000 true node_1 9000 node_2 9000 ' > /etc/clickhouse-server/config.d/remote_servers.xml] Connection dropped: socket connection error: None Executing query SYSTEM RELOAD CONFIG on node [gw7] PASSED test_reload_clusters_config/test.py::test_add_cluster test_reload_clusters_config/test.py::test_delete_cluster Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query select status in ('Running', 'RunningOnAnotherReplica') from system.view_refreshes where view = 'f' on node2 Executing query select table, uuid from system.tables where database = 're' on node1 Executing query select count() from system.zookeeper where path = '/clickhouse/tables/7c4b9a76-9ee6-4e0c-b525-643ab6bc90f4' and name = 'shard1' on node1 Executing query select count() from system.zookeeper where path = '/clickhouse/tables/eab02509-37e9-4ea5-b358-eda6df49082c' and name = 'shard1' on node1 Executing query select count() from system.zookeeper where path = '/clickhouse/tables/4133000a-6e88-4d83-8484-6e35dd4e74cf' and name = 'shard1' on node2 Executing query select count() from system.zookeeper where path = '/clickhouse/tables/79f6eb79-b3f4-41d7-9550-995c5554eec9' and name = 'shard1' on node2 Executing query select count() from system.zookeeper where path = '/clickhouse/tables/bb93da76-d3e6-4051-b0db-d28b930a2a67' and name = 'shard1' on node2 Executing query select count() from system.zookeeper where path = '/clickhouse/tables/ae79fdb2-e7d7-45ef-ab10-b7b2795245fb' and name = 'shard1' on node1 Executing query drop table re.a on node1 Executing query select count() from system.zookeeper where path = '/clickhouse/tables/7c4b9a76-9ee6-4e0c-b525-643ab6bc90f4' and name = 'shard1' on node1 Executing query drop table re.append on node2 Executing query select count() from system.zookeeper where path = '/clickhouse/tables/eab02509-37e9-4ea5-b358-eda6df49082c' and name = 'shard1' on node1 Executing query drop table re.append_uncoordinated sync on node2 Executing query select count() from system.zookeeper where path = '/clickhouse/tables/4133000a-6e88-4d83-8484-6e35dd4e74cf' and name = 'shard1' on node1 Executing query drop table re.e on node1 Executing query select count() from system.zookeeper where path = '/clickhouse/tables/79f6eb79-b3f4-41d7-9550-995c5554eec9' and name = 'shard1' on node1 Executing query drop table re.f sync on node1 Executing query select count() from system.zookeeper where path = '/clickhouse/tables/bb93da76-d3e6-4051-b0db-d28b930a2a67' and name = 'shard1' on node2 Executing query drop table re.unreplicated_uncoordinated on node2 Executing query select count() from system.zookeeper where path = '/clickhouse/tables/ae79fdb2-e7d7-45ef-ab10-b7b2795245fb' and name = 'shard1' on node2 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select 1 as x on node2 Executing query drop table re.g on node1 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select 1 as x on node1 Executing query drop table re.g on node1 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select 1 as x on node1 Executing query drop table re.g on node2 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select 1 as x on node2 Executing query drop table re.g on node1 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select 1 as x on node2 Executing query drop table re.g on node1 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select 1 as x on node1 Executing query drop table re.g on node1 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x empty as select 1 as x on node2 Executing query drop table re.g on node2 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select 1 as x on node2 Executing query drop table re.g on node1 Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x empty as select 1 as x on node1 Executing query drop table re.g on node2 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select 1 as x on node1 Executing query drop table re.g on node1 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select 1 as x on node2 Executing query drop table re.g on node1 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x empty as select 1 as x on node1 Executing query drop table re.g on node2 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x empty as select 1 as x on node2 Executing query drop table re.g on node2 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x empty as select 1 as x on node2 Executing query drop table re.g on node1 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select 1 as x on node2 Executing query drop table re.g on node1 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select 1 as x on node1 Executing query drop table re.g on node2 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select 1 as x on node1 Connection dropped: socket connection error: None Executing query drop table re.g on node2 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select 1 as x on node2 Executing query drop table re.g on node1 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x as select 1 as x on node1 Executing query drop table re.g on node2 Executing query create materialized view re.g refresh every 1 second (x Int64) engine ReplicatedMergeTree order by x empty as select 1 as x on node2 Executing query drop table re.g on node2 Executing query show tables from re on node1 Executing query show tables from re on node2 Executing query drop database re sync on node1 Executing query drop database re sync on node2 Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node [gw5] PASSED test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db test_refreshable_mv/test.py::test_refreshable_mv_in_system_db Executing query create materialized view system.a refresh every 1 second (x Int64) engine Memory as select number+1 as x from numbers(2);system refresh view system.a; on node1 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c ps -C clickhouse] Stdout: PID TTY TIME CMD Stdout: 9538 ? 00:00:05 clickhouse run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] Command:[docker exec -u root roottestrefreshablemv-gw5-node1-1 bash -c pkill clickhouse] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:9538 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:9538 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:9538 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:9538 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:9538 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:9538 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:9538 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:9538 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] No clickhouse process running. Start new one. http://localhost:None "POST /v1.46/containers/roottestrefreshablemv-gw5-node1-1/exec HTTP/1.1" 201 74 http://localhost:None "POST /v1.46/exec/2057933138f46c93e203ed9c7cac37bf1d8dd5fabe8ad3c6dc906af6044300af/start HTTP/1.1" 200 0 http://localhost:None "GET /v1.46/exec/2057933138f46c93e203ed9c7cac37bf1d8dd5fabe8ad3c6dc906af6044300af/json HTTP/1.1" 200 587 run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:10405 Clickhouse process running. run container_id:roottestrefreshablemv-gw5-node1-1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] Command:[docker exec roottestrefreshablemv-gw5-node1-1 bash -c ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'] Stdout:10405 Executing query select 20 on node1 Executing query system refresh view system.a on node1 Connection dropped: socket connection error: None Executing query select count(), sum(x) from system.a on node1 Executing query drop table system.a on node1 Command:[docker compose --env-file /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/.env --project-name roottestrefreshablemv-gw5 --file /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node2/docker-compose.yml stop --timeout 20] [gw5] PASSED test_refreshable_mv/test.py::test_refreshable_mv_in_system_db Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Stderr: Container roottestrefreshablemv-gw5-node1-1 Stopping Stderr: Container roottestrefreshablemv-gw5-node2-1 Stopping Stderr: Container roottestrefreshablemv-gw5-node1-1 Stopped Stderr: Container roottestrefreshablemv-gw5-node2-1 Stopped Stderr: Container roottestrefreshablemv-gw5-zoo3-1 Stopping Stderr: Container roottestrefreshablemv-gw5-zoo1-1 Stopping Stderr: Container roottestrefreshablemv-gw5-zoo2-1 Stopping Stderr: Container roottestrefreshablemv-gw5-zoo3-1 Stopped Stderr: Container roottestrefreshablemv-gw5-zoo1-1 Stopped Stderr: Container roottestrefreshablemv-gw5-zoo2-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/.env --project-name roottestrefreshablemv-gw5 --file /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_refreshable_mv/_instances-0-gw5/node2/docker-compose.yml down --volumes] Stderr: Container roottestrefreshablemv-gw5-node1-1 Stopping Stderr: Container roottestrefreshablemv-gw5-node2-1 Stopping Stderr: Container roottestrefreshablemv-gw5-node1-1 Stopped Stderr: Container roottestrefreshablemv-gw5-node1-1 Removing Stderr: Container roottestrefreshablemv-gw5-node2-1 Stopped Stderr: Container roottestrefreshablemv-gw5-node2-1 Removing Stderr: Container roottestrefreshablemv-gw5-node1-1 Removed Stderr: Container roottestrefreshablemv-gw5-node2-1 Removed Stderr: Container roottestrefreshablemv-gw5-zoo1-1 Stopping Stderr: Container roottestrefreshablemv-gw5-zoo2-1 Stopping Stderr: Container roottestrefreshablemv-gw5-zoo3-1 Stopping Stderr: Container roottestrefreshablemv-gw5-zoo2-1 Stopped Stderr: Container roottestrefreshablemv-gw5-zoo2-1 Removing Stderr: Container roottestrefreshablemv-gw5-zoo1-1 Stopped Stderr: Container roottestrefreshablemv-gw5-zoo1-1 Removing Stderr: Container roottestrefreshablemv-gw5-zoo3-1 Stopped Stderr: Container roottestrefreshablemv-gw5-zoo3-1 Removing Stderr: Container roottestrefreshablemv-gw5-zoo2-1 Removed Stderr: Container roottestrefreshablemv-gw5-zoo1-1 Removed Stderr: Container roottestrefreshablemv-gw5-zoo3-1 Removed Stderr: Network roottestrefreshablemv-gw5_default Removing Stderr: Network roottestrefreshablemv-gw5_default Removed Cleanup called Docker networks for project roottestrefreshablemv-gw5 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestrefreshablemv-gw5 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestrefreshablemv-gw5 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestrefreshablemv-gw5-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestrefreshablemv-gw5 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Connection dropped: socket connection error: None Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Connection dropped: socket connection error: None Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Connection dropped: socket connection error: None Connection dropped: socket connection error: None Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Connection dropped: socket connection error: None Failed connecting to Zookeeper within the connection retry policy. Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Connection dropped: socket connection error: None Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node [gw7] FAILED test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_simple_reload Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Connection dropped: socket connection error: None Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Connection dropped: socket connection error: None Failed connecting to Zookeeper within the connection retry policy. Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node Executing query SYSTEM RELOAD CONFIG on node Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node [gw7] PASSED test_reload_clusters_config/test.py::test_simple_reload test_reload_clusters_config/test.py::test_update_one_cluster Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Connection dropped: socket connection error: None Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Connection dropped: socket connection error: None Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/.env --project-name roottestreloadclustersconfig-gw7 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop --timeout 20] [gw7] FAILED test_reload_clusters_config/test.py::test_update_one_cluster Stderr: Container roottestreloadclustersconfig-gw7-node-1 Stopping Stderr: Container roottestreloadclustersconfig-gw7-node-1 Stopped Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Stopping Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Stopping Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Stopping Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Stopped Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Stopped Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/.env --project-name roottestreloadclustersconfig-gw7 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml down --volumes] Stderr: Container roottestreloadclustersconfig-gw7-node-1 Stopping Stderr: Container roottestreloadclustersconfig-gw7-node-1 Stopped Stderr: Container roottestreloadclustersconfig-gw7-node-1 Removing Stderr: Container roottestreloadclustersconfig-gw7-node-1 Removed Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Stopping Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Stopping Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Stopping Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Stopped Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Removing Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Stopped Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Removing Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Stopped Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Removing Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Removed Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Removed Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Removed Stderr: Network roottestreloadclustersconfig-gw7_default Removing Stderr: Network roottestreloadclustersconfig-gw7_default Removed Cleanup called Docker networks for project roottestreloadclustersconfig-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreloadclustersconfig-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreloadclustersconfig-gw7 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreloadclustersconfig-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreloadclustersconfig-gw7 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 =================================== FAILURES =================================== _____________________________ test_delete_cluster ______________________________ [gw7] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_delete_cluster(started_cluster): send_repeated_query("distributed") send_repeated_query("distributed2") > assert get_errors_count("test_cluster") > 0 E AssertionError: assert 0 > 0 E + where 0 = get_errors_count('test_cluster') test_reload_clusters_config/test.py:217: AssertionError ------------------------------ Captured log call ------------------------------- 2025-04-02 04:00:50 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:01:02 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:01:14 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:01:26 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:01:39 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:01:51 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:02:04 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:02:17 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:02:29 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:02:42 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:02:54 [ 719 ] DEBUG : Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node (cluster.py:3564, query) ___________________________ test_update_one_cluster ____________________________ [gw7] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_update_one_cluster(started_cluster): send_repeated_query("distributed") send_repeated_query("distributed2") > assert get_errors_count("test_cluster") > 0 E AssertionError: assert 0 > 0 E + where 0 = get_errors_count('test_cluster') test_reload_clusters_config/test.py:202: AssertionError ------------------------------ Captured log call ------------------------------- 2025-04-02 04:03:56 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:04:10 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:04:23 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:04:35 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:04:47 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:04:59 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:05:11 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:05:23 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:05:35 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:05:47 [ 719 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3647, query_and_get_error) 2025-04-02 04:05:59 [ 719 ] DEBUG : Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node (cluster.py:3564, query) ---------------------------- Captured log teardown ----------------------------- 2025-04-02 04:05:59 [ 719 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/.env --project-name roottestreloadclustersconfig-gw7 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop --timeout 20] (cluster.py:120, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-node-1 Stopping (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-node-1 Stopped (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Stopping (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Stopping (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Stopping (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Stopped (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Stopped (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Stopped (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:120, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/.env --project-name roottestreloadclustersconfig-gw7 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-0-gw7/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml down --volumes] (cluster.py:120, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-node-1 Stopping (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-node-1 Stopped (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-node-1 Removing (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-node-1 Removed (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Stopping (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Stopping (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Stopping (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Stopped (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Removing (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Stopped (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Removing (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Stopped (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Removing (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo2-1 Removed (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo3-1 Removed (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw7-zoo1-1 Removed (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Network roottestreloadclustersconfig-gw7_default Removing (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stderr: Network roottestreloadclustersconfig-gw7_default Removed (cluster.py:146, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Cleanup called (cluster.py:876, cleanup) 2025-04-02 04:06:07 [ 719 ] DEBUG : Docker networks for project roottestreloadclustersconfig-gw7 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2025-04-02 04:06:07 [ 719 ] DEBUG : Docker containers for project roottestreloadclustersconfig-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2025-04-02 04:06:07 [ 719 ] DEBUG : Docker volumes for project roottestreloadclustersconfig-gw7 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2025-04-02 04:06:07 [ 719 ] DEBUG : Command:[docker container list --all --filter name='^/roottestreloadclustersconfig-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:120, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Unstopped containers: {} (cluster.py:890, cleanup) 2025-04-02 04:06:07 [ 719 ] DEBUG : No running containers for project: roottestreloadclustersconfig-gw7 (cluster.py:904, cleanup) 2025-04-02 04:06:07 [ 719 ] DEBUG : Trying to prune unused networks... (cluster.py:910, cleanup) 2025-04-02 04:06:07 [ 719 ] DEBUG : Trying to prune unused images... (cluster.py:926, cleanup) 2025-04-02 04:06:07 [ 719 ] DEBUG : Command:[docker image prune -f] (cluster.py:120, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:144, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Images pruned (cluster.py:929, cleanup) 2025-04-02 04:06:07 [ 719 ] DEBUG : Trying to prune unused volumes... (cluster.py:935, cleanup) 2025-04-02 04:06:07 [ 719 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:120, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Stdout:1 (cluster.py:144, run_and_check) 2025-04-02 04:06:07 [ 719 ] DEBUG : Volumes pruned: 1 (cluster.py:940, cleanup) ============================== slowest durations =============================== 138.26s call test_reload_clusters_config/test.py::test_add_cluster 124.37s call test_reload_clusters_config/test.py::test_delete_cluster 123.38s call test_reload_clusters_config/test.py::test_update_one_cluster 118.75s call test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke 61.25s call test_reload_clusters_config/test.py::test_simple_reload 34.85s call test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db 28.97s setup test_replicated_zero_copy_projection_mutation/test.py::test_all_projection_files_are_dropped_when_part_is_dropped 27.51s call test_recompression_ttl/test.py::test_recompression_multiple_ttls 26.42s setup test_reload_clusters_config/test.py::test_add_cluster 26.00s setup test_read_only_table/test.py::test_restart_zookeeper 22.95s call test_recompression_ttl/test.py::test_recompression_simple 21.97s call test_role/test.py::test_roles_cache 21.90s teardown test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 21.79s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format 21.73s teardown test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header] 21.72s teardown test_replicated_zero_copy_projection_mutation/test.py::test_hardlinks_preserved_when_projection_dropped 20.77s setup test_restore_replica/test.py::test_restore_replica_alive_replicas 20.72s setup test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] 20.47s call test_postgresql_database_engine/test.py::test_inaccessible_postgresql_database_engine_filterable_on_system_tables 19.88s setup test_s3_cluster/test.py::test_ambiguous_join 19.65s teardown test_restore_replica/test.py::test_restore_replica_sequential 18.88s call test_recompression_ttl/test.py::test_recompression_replicated 18.72s setup test_replication_credentials/test.py::test_credentials_and_no_credentials 17.02s call test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] 16.83s setup test_replication_without_zookeeper/test.py::test_startup_without_zookeeper 16.77s setup test_prometheus_protocols/test.py::test_64bit_id 16.09s setup test_recompression_ttl/test.py::test_recompression_multiple_ttls 15.87s setup test_replica_is_active/test.py::test_replica_is_active 14.17s setup test_postgresql_database_engine/test.py::test_datetime 13.65s teardown test_replicating_constants/test.py::test_different_versions 13.33s setup test_reloading_settings_from_users_xml/test.py::test_force_reload 13.25s call test_replicated_zero_copy_projection_mutation/test.py::test_hardlinks_preserved_when_projection_dropped 12.93s call test_read_only_table/test.py::test_restart_zookeeper 12.63s setup test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order 12.21s setup test_restart_server/test.py::test_drop_memory_database 12.18s setup test_role/test.py::test_admin_option 12.13s setup test_reload_certificate/test.py::test_ECcert_reload 11.83s setup test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header] 11.46s teardown test_replication_credentials/test.py::test_same_credentials 11.18s call test_recovery_time_metric/test.py::test_recovery_time_metric 11.11s setup test_rocksdb_read_only/test.py::test_dirctory_missing_after_stop 11.00s setup test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke 10.89s call test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers 10.78s setup test_replicating_constants/test.py::test_different_versions 10.67s setup test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable 10.47s call test_replication_without_zookeeper/test.py::test_startup_without_zookeeper 10.32s setup test_relative_filepath/test.py::test_filepath 10.04s call test_refreshable_mv/test.py::test_refreshable_mv_in_system_db 9.97s setup test_recovery_time_metric/test.py::test_recovery_time_metric 8.85s call test_rocksdb_read_only/test.py::test_dirctory_missing_after_stop 8.24s teardown test_reload_clusters_config/test.py::test_update_one_cluster 7.74s setup test_replica_can_become_leader/test.py::test_can_become_leader 7.64s teardown test_read_only_table/test.py::test_restart_zookeeper 7.40s call test_replica_is_active/test.py::test_replica_is_active 7.35s setup test_render_log_file_name_templates/test.py::test_check_file_names 7.25s call test_role/test.py::test_role_expiration[True] 7.00s teardown test_prometheus_endpoint/test.py::test_prometheus_endpoint 6.74s setup test_prometheus_endpoint/test.py::test_prometheus_endpoint 6.66s teardown test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers 6.33s setup test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers 6.27s teardown test_recompression_ttl/test.py::test_recompression_simple 6.03s call test_role/test.py::test_role_expiration[False] 5.68s teardown test_replica_is_active/test.py::test_replica_is_active 5.68s teardown test_replica_can_become_leader/test.py::test_can_become_leader 5.18s teardown test_refreshable_mv/test.py::test_refreshable_mv_in_system_db 5.15s teardown test_render_log_file_name_templates/test.py::test_check_file_names 4.63s call test_restart_server/test.py::test_flushes_async_insert_queue 4.44s teardown test_postgresql_database_engine/test.py::test_predefined_connection_configuration 4.37s call test_restore_replica/test.py::test_restore_replica_sequential 4.25s teardown test_prometheus_protocols/test.py::test_tags_to_columns 4.15s teardown test_reload_certificate/test.py::test_first_than_second_cert 4.15s call test_prometheus_protocols/test.py::test_default 4.14s call test_prometheus_protocols/test.py::test_inner_engines 3.91s teardown test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order 3.85s call test_reloading_settings_from_users_xml/test.py::test_reload_on_timeout 3.82s teardown test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 3.65s call test_restore_replica/test.py::test_restore_replica_alive_replicas 3.63s call test_restore_replica/test.py::test_restore_replica_parallel 3.61s call test_prometheus_protocols/test.py::test_external_tables 3.59s call test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable 3.39s teardown test_role/test.py::test_set_role 3.27s teardown test_reloading_settings_from_users_xml/test.py::test_unknown_setting_reload_on_timeout 2.98s teardown test_relative_filepath/test.py::test_filepath 2.94s call test_replication_credentials/test.py::test_credentials_and_no_credentials 2.71s call test_restart_server/test.py::test_drop_memory_database 2.70s call test_role/test.py::test_introspection 2.69s call test_replication_credentials/test.py::test_different_credentials 2.55s call test_replication_credentials/test.py::test_same_credentials 2.51s teardown test_replication_without_zookeeper/test.py::test_startup_without_zookeeper 2.43s setup test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 2.40s call test_replication_credentials/test.py::test_no_credentials 2.31s call test_prometheus_protocols/test.py::test_tags_to_columns 2.11s call test_prometheus_protocols/test.py::test_custom_id_algorithm 1.98s call test_role/test.py::test_grant_role_to_role 1.94s call test_role/test.py::test_revoke_requires_admin_option 1.55s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] 1.55s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] 1.53s call test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 1.48s call test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 1.48s call test_postgresql_database_engine/test.py::test_postgresql_database_engine_table_cache 1.45s call test_postgresql_database_engine/test.py::test_predefined_connection_configuration 1.39s call test_role/test.py::test_admin_option 1.34s call test_prometheus_protocols/test.py::test_64bit_id 1.30s teardown test_restart_server/test.py::test_flushes_async_insert_queue 1.30s call test_reloading_settings_from_users_xml/test.py::test_unknown_setting_reload_on_timeout 1.26s teardown test_rocksdb_read_only/test.py::test_read_only 1.21s teardown test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable 1.16s call test_prometheus_protocols/test.py::test_create_as_table 1.15s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] 1.15s teardown test_recovery_time_metric/test.py::test_recovery_time_metric 1.14s call test_postgresql_database_engine/test.py::test_postgresql_database_engine_with_clickhouse_ddl 1.05s call test_rocksdb_read_only/test.py::test_read_only 1.05s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] 1.01s call test_prometheus_endpoint/test.py::test_prometheus_endpoint 0.84s call test_s3_cluster/test.py::test_cluster_default_expression 0.83s call test_postgresql_database_engine/test.py::test_postgresql_database_with_schema 0.82s call test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header] 0.82s call test_reload_certificate/test.py::test_cert_with_pass_phrase 0.76s call test_reloading_settings_from_users_xml/test.py::test_force_reload 0.75s call test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format 0.72s call test_postgresql_database_engine/test.py::test_postgresql_database_engine_queries 0.71s call test_role/test.py::test_combine_privileges 0.69s call test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header] 0.66s call test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header] 0.64s call test_role/test.py::test_create_role 0.63s call test_s3_cluster/test.py::test_cluster_format_detection 0.62s call test_role/test.py::test_function_current_roles 0.61s call test_reload_certificate/test.py::test_chain_reload 0.59s setup test_replication_credentials/test.py::test_same_credentials 0.56s call test_replicated_zero_copy_projection_mutation/test.py::test_all_projection_files_are_dropped_when_part_is_dropped 0.55s call test_reload_certificate/test.py::test_ECcert_reload 0.53s call test_postgresql_database_engine/test.py::test_postgresql_password_leak 0.52s call test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_enum 0.51s teardown test_prometheus_protocols/test.py::test_64bit_id 0.50s teardown test_prometheus_protocols/test.py::test_create_as_table 0.47s call test_s3_cluster/test.py::test_ambiguous_join 0.47s call test_postgresql_database_engine/test.py::test_postgres_database_engine_with_postgres_ddl 0.43s call test_postgresql_database_engine/test.py::test_get_create_table_query_with_multidim_arrays 0.43s call test_reload_certificate/test.py::test_first_than_second_cert 0.43s call test_role/test.py::test_changing_default_roles_affects_new_sessions_only 0.42s teardown test_prometheus_protocols/test.py::test_inner_engines 0.39s call test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 0.39s call test_postgresql_database_engine/test.py::test_datetime 0.37s teardown test_prometheus_protocols/test.py::test_read_auth 0.36s call test_s3_cluster/test.py::test_cluster_with_header 0.35s call test_s3_cluster/test.py::test_cluster_with_named_collection 0.34s call test_relative_filepath/test.py::test_filepath 0.33s call test_replica_can_become_leader/test.py::test_can_become_leader 0.33s teardown test_role/test.py::test_role_expiration[True] 0.33s teardown test_prometheus_protocols/test.py::test_default 0.33s teardown test_prometheus_protocols/test.py::test_external_tables 0.33s teardown test_prometheus_protocols/test.py::test_remote_write_v1_status_code 0.31s call test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_int 0.30s call test_reloading_settings_from_users_xml/test.py::test_unknown_setting_force_reload 0.29s teardown test_prometheus_protocols/test.py::test_custom_id_algorithm 0.29s call test_render_log_file_name_templates/test.py::test_check_file_names 0.25s call test_postgresql_database_engine/test.py::test_postgres_database_old_syntax 0.25s call test_role/test.py::test_set_role 0.23s call test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order 0.23s call test_s3_cluster/test.py::test_count_macro 0.22s setup test_reloading_settings_from_users_xml/test.py::test_unknown_setting_reload_on_timeout 0.20s teardown test_role/test.py::test_roles_cache 0.19s setup test_replication_credentials/test.py::test_different_credentials 0.19s setup test_replication_credentials/test.py::test_no_credentials 0.18s call test_postgresql_database_engine/test.py::test_postgresql_fetch_tables 0.18s teardown test_role/test.py::test_admin_option 0.18s teardown test_role/test.py::test_role_expiration[False] 0.18s call test_s3_cluster/test.py::test_count 0.18s teardown test_role/test.py::test_combine_privileges 0.16s call test_restore_replica/test.py::test_restore_replica_invalid_tables 0.16s setup test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_enum 0.13s teardown test_role/test.py::test_introspection 0.13s teardown test_role/test.py::test_create_role 0.13s teardown test_role/test.py::test_revoke_requires_admin_option 0.13s teardown test_role/test.py::test_function_current_roles 0.13s teardown test_role/test.py::test_changing_default_roles_affects_new_sessions_only 0.12s call test_replicating_constants/test.py::test_different_versions 0.11s setup test_reloading_settings_from_users_xml/test.py::test_reload_on_timeout 0.11s setup test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_int 0.11s setup test_reloading_settings_from_users_xml/test.py::test_unknown_setting_force_reload 0.10s teardown test_role/test.py::test_grant_role_to_role 0.08s call test_prometheus_protocols/test.py::test_remote_write_v1_status_code 0.07s call test_prometheus_protocols/test.py::test_read_auth 0.00s setup test_replicated_zero_copy_projection_mutation/test.py::test_hardlinks_preserved_when_projection_dropped 0.00s setup test_s3_cluster/test.py::test_cluster_format_detection 0.00s setup test_postgresql_database_engine/test.py::test_postgresql_fetch_tables 0.00s teardown test_restore_replica/test.py::test_restore_replica_parallel 0.00s teardown test_restore_replica/test.py::test_restore_replica_alive_replicas 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] 0.00s teardown test_s3_cluster/test.py::test_ambiguous_join 0.00s teardown test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] 0.00s setup test_role/test.py::test_grant_role_to_role 0.00s teardown test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header] 0.00s teardown test_replicated_zero_copy_projection_mutation/test.py::test_all_projection_files_are_dropped_when_part_is_dropped 0.00s setup test_prometheus_protocols/test.py::test_tags_to_columns 0.00s setup test_role/test.py::test_function_current_roles 0.00s setup test_prometheus_protocols/test.py::test_inner_engines 0.00s setup test_prometheus_protocols/test.py::test_read_auth 0.00s setup test_prometheus_protocols/test.py::test_default 0.00s teardown test_reloading_settings_from_users_xml/test.py::test_unknown_setting_force_reload 0.00s setup test_prometheus_protocols/test.py::test_remote_write_v1_status_code 0.00s setup test_prometheus_protocols/test.py::test_create_as_table 0.00s setup test_reload_clusters_config/test.py::test_delete_cluster 0.00s setup test_role/test.py::test_introspection 0.00s setup test_role/test.py::test_role_expiration[True] 0.00s setup test_reload_certificate/test.py::test_cert_with_pass_phrase 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] 0.00s setup test_reload_certificate/test.py::test_chain_reload 0.00s teardown test_recompression_ttl/test.py::test_recompression_multiple_ttls 0.00s setup test_role/test.py::test_combine_privileges 0.00s setup test_role/test.py::test_revoke_requires_admin_option 0.00s teardown test_reload_clusters_config/test.py::test_add_cluster 0.00s setup test_role/test.py::test_create_role 0.00s setup test_role/test.py::test_roles_cache 0.00s setup test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header] 0.00s setup test_role/test.py::test_changing_default_roles_affects_new_sessions_only 0.00s setup test_restore_replica/test.py::test_restore_replica_invalid_tables 0.00s setup test_role/test.py::test_role_expiration[False] 0.00s teardown test_rocksdb_read_only/test.py::test_dirctory_missing_after_stop 0.00s teardown test_replication_credentials/test.py::test_credentials_and_no_credentials 0.00s setup test_restore_replica/test.py::test_restore_replica_sequential 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] 0.00s setup test_postgresql_database_engine/test.py::test_postgres_database_old_syntax 0.00s setup test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header] 0.00s setup test_postgresql_database_engine/test.py::test_inaccessible_postgresql_database_engine_filterable_on_system_tables 0.00s setup test_prometheus_protocols/test.py::test_external_tables 0.00s setup test_role/test.py::test_set_role 0.00s setup test_s3_cluster/test.py::test_cluster_default_expression 0.00s setup test_postgresql_database_engine/test.py::test_postgresql_database_engine_queries 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] 0.00s setup test_recompression_ttl/test.py::test_recompression_replicated 0.00s setup test_refreshable_mv/test.py::test_refreshable_mv_in_system_db 0.00s setup test_s3_cluster/test.py::test_cluster_with_header 0.00s setup test_prometheus_protocols/test.py::test_custom_id_algorithm 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] 0.00s setup test_s3_cluster/test.py::test_count 0.00s setup test_postgresql_database_engine/test.py::test_postgresql_password_leak 0.00s setup test_reload_clusters_config/test.py::test_update_one_cluster 0.00s teardown test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db 0.00s setup test_rocksdb_read_only/test.py::test_read_only 0.00s teardown test_postgresql_database_engine/test.py::test_datetime 0.00s setup test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db 0.00s setup test_postgresql_database_engine/test.py::test_postgres_database_engine_with_postgres_ddl 0.00s setup test_postgresql_database_engine/test.py::test_predefined_connection_configuration 0.00s setup test_postgresql_database_engine/test.py::test_get_create_table_query_with_multidim_arrays 0.00s teardown test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_int 0.00s teardown test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_enum 0.00s setup test_recompression_ttl/test.py::test_recompression_simple 0.00s setup test_s3_cluster/test.py::test_cluster_with_named_collection 0.00s setup test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 0.00s teardown test_postgresql_database_engine/test.py::test_postgresql_database_with_schema 0.00s setup test_postgresql_database_engine/test.py::test_postgresql_database_engine_with_clickhouse_ddl 0.00s setup test_reload_clusters_config/test.py::test_simple_reload 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] 0.00s teardown test_postgresql_database_engine/test.py::test_get_create_table_query_with_multidim_arrays 0.00s setup test_reload_certificate/test.py::test_first_than_second_cert 0.00s teardown test_replication_credentials/test.py::test_no_credentials 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 0.00s teardown test_reloading_settings_from_users_xml/test.py::test_force_reload 0.00s teardown test_reload_certificate/test.py::test_ECcert_reload 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] 0.00s teardown test_replication_credentials/test.py::test_different_credentials 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format 0.00s teardown test_s3_cluster/test.py::test_cluster_with_named_collection 0.00s setup test_postgresql_database_engine/test.py::test_postgresql_database_engine_table_cache 0.00s teardown test_postgresql_database_engine/test.py::test_postgres_database_engine_with_postgres_ddl 0.00s teardown test_s3_cluster/test.py::test_cluster_format_detection 0.00s setup test_postgresql_database_engine/test.py::test_postgresql_database_with_schema 0.00s teardown test_reload_certificate/test.py::test_cert_with_pass_phrase 0.00s teardown test_reloading_settings_from_users_xml/test.py::test_reload_on_timeout 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] 0.00s teardown test_restart_server/test.py::test_drop_memory_database 0.00s teardown test_s3_cluster/test.py::test_cluster_default_expression 0.00s setup test_restore_replica/test.py::test_restore_replica_parallel 0.00s teardown test_postgresql_database_engine/test.py::test_postgresql_database_engine_table_cache 0.00s setup test_restart_server/test.py::test_flushes_async_insert_queue 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] 0.00s teardown test_reload_certificate/test.py::test_chain_reload 0.00s setup test_s3_cluster/test.py::test_count_macro 0.00s teardown test_postgresql_database_engine/test.py::test_postgresql_database_engine_queries 0.00s teardown test_reload_clusters_config/test.py::test_delete_cluster 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] 0.00s teardown test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header] 0.00s teardown test_recompression_ttl/test.py::test_recompression_replicated 0.00s teardown test_postgresql_database_engine/test.py::test_inaccessible_postgresql_database_engine_filterable_on_system_tables 0.00s teardown test_postgresql_database_engine/test.py::test_postgresql_database_engine_with_clickhouse_ddl 0.00s teardown test_postgresql_database_engine/test.py::test_postgres_database_old_syntax 0.00s teardown test_postgresql_database_engine/test.py::test_postgresql_password_leak 0.00s teardown test_postgresql_database_engine/test.py::test_postgresql_fetch_tables 0.00s teardown test_s3_cluster/test.py::test_cluster_with_header 0.00s teardown test_reload_clusters_config/test.py::test_simple_reload 0.00s setup test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 0.00s teardown test_s3_cluster/test.py::test_count_macro 0.00s call test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] 0.00s teardown test_s3_cluster/test.py::test_count 0.00s teardown test_restore_replica/test.py::test_restore_replica_invalid_tables 0.00s call test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] 0.00s teardown test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] =========================== short test summary info ============================ FAILED test_reload_clusters_config/test.py::test_delete_cluster - AssertionEr... FAILED test_reload_clusters_config/test.py::test_update_one_cluster - Asserti... PASSED test_reload_certificate/test.py::test_ECcert_reload PASSED test_reload_certificate/test.py::test_cert_with_pass_phrase PASSED test_role/test.py::test_admin_option PASSED test_reloading_settings_from_users_xml/test.py::test_force_reload PASSED test_reload_certificate/test.py::test_chain_reload PASSED test_role/test.py::test_changing_default_roles_affects_new_sessions_only PASSED test_reload_certificate/test.py::test_first_than_second_cert PASSED test_postgresql_database_engine/test.py::test_datetime PASSED test_postgresql_database_engine/test.py::test_get_create_table_query_with_multidim_arrays PASSED test_role/test.py::test_combine_privileges PASSED test_role/test.py::test_create_role PASSED test_role/test.py::test_function_current_roles PASSED test_reloading_settings_from_users_xml/test.py::test_reload_on_timeout PASSED test_prometheus_protocols/test.py::test_64bit_id PASSED test_role/test.py::test_grant_role_to_role PASSED test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_enum PASSED test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_int PASSED test_reloading_settings_from_users_xml/test.py::test_unknown_setting_force_reload PASSED test_prometheus_protocols/test.py::test_create_as_table PASSED test_s3_cluster/test.py::test_ambiguous_join PASSED test_reloading_settings_from_users_xml/test.py::test_unknown_setting_reload_on_timeout PASSED test_s3_cluster/test.py::test_cluster_default_expression PASSED test_role/test.py::test_introspection PASSED test_replication_credentials/test.py::test_credentials_and_no_credentials PASSED test_s3_cluster/test.py::test_cluster_format_detection PASSED test_s3_cluster/test.py::test_cluster_with_header PASSED test_prometheus_protocols/test.py::test_custom_id_algorithm PASSED test_s3_cluster/test.py::test_cluster_with_named_collection PASSED test_s3_cluster/test.py::test_count PASSED test_s3_cluster/test.py::test_count_macro PASSED test_role/test.py::test_revoke_requires_admin_option PASSED test_restore_replica/test.py::test_restore_replica_alive_replicas PASSED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated PASSED test_replication_credentials/test.py::test_different_credentials PASSED test_restore_replica/test.py::test_restore_replica_invalid_tables PASSED test_prometheus_protocols/test.py::test_default PASSED test_replication_credentials/test.py::test_no_credentials PASSED test_restore_replica/test.py::test_restore_replica_parallel PASSED test_role/test.py::test_role_expiration[False] PASSED test_replication_credentials/test.py::test_same_credentials PASSED test_prometheus_protocols/test.py::test_external_tables PASSED test_restore_replica/test.py::test_restore_replica_sequential PASSED test_prometheus_protocols/test.py::test_inner_engines PASSED test_postgresql_database_engine/test.py::test_inaccessible_postgresql_database_engine_filterable_on_system_tables PASSED test_prometheus_protocols/test.py::test_read_auth PASSED test_postgresql_database_engine/test.py::test_postgres_database_engine_with_postgres_ddl PASSED test_postgresql_database_engine/test.py::test_postgres_database_old_syntax PASSED test_prometheus_protocols/test.py::test_remote_write_v1_status_code PASSED test_postgresql_database_engine/test.py::test_postgresql_database_engine_queries PASSED test_role/test.py::test_role_expiration[True] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain] PASSED test_postgresql_database_engine/test.py::test_postgresql_database_engine_table_cache PASSED test_prometheus_protocols/test.py::test_tags_to_columns PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0] PASSED test_postgresql_database_engine/test.py::test_postgresql_database_engine_with_clickhouse_ddl PASSED test_postgresql_database_engine/test.py::test_postgresql_database_with_schema PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1] PASSED test_postgresql_database_engine/test.py::test_postgresql_fetch_tables PASSED test_postgresql_database_engine/test.py::test_postgresql_password_leak PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2] PASSED test_postgresql_database_engine/test.py::test_predefined_connection_configuration PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4] PASSED test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format PASSED test_prometheus_endpoint/test.py::test_prometheus_endpoint PASSED test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header] PASSED test_role/test.py::test_roles_cache PASSED test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header] PASSED test_role/test.py::test_set_role PASSED test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header] PASSED test_recompression_ttl/test.py::test_recompression_multiple_ttls PASSED test_rocksdb_read_only/test.py::test_dirctory_missing_after_stop PASSED test_rocksdb_read_only/test.py::test_read_only PASSED test_range_hashed_dictionary_types/test.py::test_range_hashed_dict PASSED test_restart_server/test.py::test_drop_memory_database PASSED test_replicated_zero_copy_projection_mutation/test.py::test_all_projection_files_are_dropped_when_part_is_dropped PASSED test_restart_server/test.py::test_flushes_async_insert_queue PASSED test_relative_filepath/test.py::test_filepath PASSED test_render_log_file_name_templates/test.py::test_check_file_names PASSED test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order PASSED test_replica_can_become_leader/test.py::test_can_become_leader PASSED test_recompression_ttl/test.py::test_recompression_replicated PASSED test_replicated_zero_copy_projection_mutation/test.py::test_hardlinks_preserved_when_projection_dropped PASSED test_recovery_time_metric/test.py::test_recovery_time_metric PASSED test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable PASSED test_recompression_ttl/test.py::test_recompression_simple PASSED test_replica_is_active/test.py::test_replica_is_active PASSED test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers PASSED test_replication_without_zookeeper/test.py::test_startup_without_zookeeper PASSED test_replicating_constants/test.py::test_different_versions PASSED test_read_only_table/test.py::test_restart_zookeeper PASSED test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke PASSED test_reload_clusters_config/test.py::test_add_cluster PASSED test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db PASSED test_refreshable_mv/test.py::test_refreshable_mv_in_system_db PASSED test_reload_clusters_config/test.py::test_simple_reload =================== 2 failed, 98 passed in 484.03s (0:08:04) =================== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 528, in subprocess.check_call(cmd, shell=True, bufsize=0) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_yhqp7s --privileged --dns-search='.' --memory=30709035008 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=8b2301119731 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_postgresql_database_engine/test.py::test_datetime test_postgresql_database_engine/test.py::test_get_create_table_query_with_multidim_arrays test_postgresql_database_engine/test.py::test_inaccessible_postgresql_database_engine_filterable_on_system_tables test_postgresql_database_engine/test.py::test_postgres_database_engine_with_postgres_ddl test_postgresql_database_engine/test.py::test_postgres_database_old_syntax test_postgresql_database_engine/test.py::test_postgresql_database_engine_queries test_postgresql_database_engine/test.py::test_postgresql_database_engine_table_cache test_postgresql_database_engine/test.py::test_postgresql_database_engine_with_clickhouse_ddl test_postgresql_database_engine/test.py::test_postgresql_database_with_schema test_postgresql_database_engine/test.py::test_postgresql_fetch_tables test_postgresql_database_engine/test.py::test_postgresql_password_leak test_postgresql_database_engine/test.py::test_predefined_connection_configuration test_profile_settings_and_constraints_order/test.py::test_profile_settings_and_constraints_order test_prometheus_endpoint/test.py::test_prometheus_endpoint test_prometheus_protocols/test.py::test_64bit_id test_prometheus_protocols/test.py::test_create_as_table test_prometheus_protocols/test.py::test_custom_id_algorithm test_prometheus_protocols/test.py::test_default test_prometheus_protocols/test.py::test_external_tables test_prometheus_protocols/test.py::test_inner_engines test_prometheus_protocols/test.py::test_read_auth test_prometheus_protocols/test.py::test_remote_write_v1_status_code test_prometheus_protocols/test.py::test_tags_to_columns test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_read_only_table/test.py::test_restart_zookeeper test_recompression_ttl/test.py::test_recompression_multiple_ttls test_recompression_ttl/test.py::test_recompression_replicated test_recompression_ttl/test.py::test_recompression_simple test_recovery_time_metric/test.py::test_recovery_time_metric test_refreshable_mv/test.py::test_refresh_vs_shutdown_smoke test_refreshable_mv/test.py::test_refreshable_mv_in_replicated_db test_refreshable_mv/test.py::test_refreshable_mv_in_system_db test_relative_filepath/test.py::test_filepath test_reload_auxiliary_zookeepers/test.py::test_reload_auxiliary_zookeepers test_reload_certificate/test.py::test_ECcert_reload test_reload_certificate/test.py::test_cert_with_pass_phrase test_reload_certificate/test.py::test_chain_reload test_reload_certificate/test.py::test_first_than_second_cert test_reload_clusters_config/test.py::test_add_cluster test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_simple_reload test_reload_clusters_config/test.py::test_update_one_cluster test_reloading_settings_from_users_xml/test.py::test_force_reload test_reloading_settings_from_users_xml/test.py::test_reload_on_timeout test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_enum test_reloading_settings_from_users_xml/test.py::test_unexpected_setting_int test_reloading_settings_from_users_xml/test.py::test_unknown_setting_force_reload test_reloading_settings_from_users_xml/test.py::test_unknown_setting_reload_on_timeout 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_log_table[s3_plain]' test_remote_blobs_naming/test_backward_compatibility.py::test_read_new_format 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case0]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case1]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case2]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case3]' 'test_remote_blobs_naming/test_backward_compatibility.py::test_replicated_merge_tree[test_case4]' test_remote_blobs_naming/test_backward_compatibility.py::test_write_new_format test_render_log_file_name_templates/test.py::test_check_file_names test_replica_can_become_leader/test.py::test_can_become_leader test_replica_is_active/test.py::test_replica_is_active test_replicated_zero_copy_projection_mutation/test.py::test_all_projection_files_are_dropped_when_part_is_dropped test_replicated_zero_copy_projection_mutation/test.py::test_hardlinks_preserved_when_projection_dropped test_replicating_constants/test.py::test_different_versions test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_restart_server/test.py::test_drop_memory_database test_restart_server/test.py::test_flushes_async_insert_queue test_restore_replica/test.py::test_restore_replica_alive_replicas test_restore_replica/test.py::test_restore_replica_invalid_tables test_restore_replica/test.py::test_restore_replica_parallel test_restore_replica/test.py::test_restore_replica_sequential test_rocksdb_read_only/test.py::test_dirctory_missing_after_stop test_rocksdb_read_only/test.py::test_read_only test_role/test.py::test_admin_option test_role/test.py::test_changing_default_roles_affects_new_sessions_only test_role/test.py::test_combine_privileges test_role/test.py::test_create_role test_role/test.py::test_function_current_roles test_role/test.py::test_grant_role_to_role test_role/test.py::test_introspection test_role/test.py::test_revoke_requires_admin_option 'test_role/test.py::test_role_expiration[False]' 'test_role/test.py::test_role_expiration[True]' test_role/test.py::test_roles_cache test_role/test.py::test_set_role test_runtime_configurable_cache_size/test.py::test_query_cache_size_is_runtime_configurable 'test_s3_access_headers/test.py::test_custom_access_header[test_access_key_id_overrides_access_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header]' 'test_s3_access_headers/test.py::test_custom_access_header[test_named_coll_overrides_access_header]' test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated -vvv -ss" altinityinfra/integration-tests-runner:2165613c5fcd ' returned non-zero exit status 1.